Fluke 289 not quite sorted.

[Towlerar]

I have a Fluke 287 I acquired from eBay for cheap. At first I found that none of the power rails would come up and pressing the power button resulted in the LED coming on for a couple seconds, flashing briefly and emitting a quick chirp, then powering down with nothing on the LCD.

I found two shorted tantalum capacitors in the 20V section for the LCD bias (C97 and C98). These caps also appeared to have shorted the output driver in the 20V converter IC (U21). Removing all of these caused the meter to  now stay on after turning on but the LED never goes out so it is still not booting correctly. There is no SPI communication between the MXS chip and the MSP430. The UART on the MXS does not output anything at all, nothing on the screen, no buttons respond. The crystal on the MSP is working fine but the crystal on the MXS is running at 100khz even after replacing.

I have come to the conclusion that the MXS chip is one sick puppy. It is getting slightly warm to the touch and indeed the thermal camera confirms this. I am powering the VBatt supply and the 20V supply from a lab supply. The 20V supply pulls about 10mA during powerup which seems normal. The 9V VBatt supply was drawing a whopping 258mA prior to finding and removing shorted components in the 20V section. Now it's at 50mA, which is still high. It seems normal is around 15-20mA on good meters according to earlier posts.

I think my next step is to obtain a JTAG programmer and upload the hex files from the existing application processor. It's unclear to me of there even is any programming in the MXS however. Can anyone confirm this? It seems maybe I could simply replace the MXS chip and it will upload the code it needs from the NOR flash? Either way, replacing the MXS chip seems to be my next obvious step.

[Towlerar]

This evening I decided to try pushing on the PCB behind the MXS processor during booting and it got into the boot menu and complained about the date and time due to a bad supercapacitor which I removed. However, the meter never fully booted because no readings made it to the screen. I wasn't sure whether the issue was with a cracked ball on the BGA or a cracked MLCC so I started measuring across the various bypass caps and found C76 would intermittently short when I pressed on the board and released. The problem seemed to go away when I removed the cap but now I can't get the meter to boot anymore. Thinking maybe the problem is actually the BGA I tried reflowing it with the hot air rework station to no avail. So it seems the MXS chip is alive but perhaps there is a cracked solder ball that doesn't want to reflow. I may have no choice but to replace it anyway. Even still, I'm concerned the meter never fully booted, which might mean a missing or corrupted file in flash.

[Towlerar]

This morning I replaced the suspected bad capacitor with a new one and I can consistently boot to the point that the power LED extinguishes and the buttons seem to mostly respond. Though, notably, the backlight button does not respond and I get nothing on the LCD anymore. This is probably no coincidence because both are tied to the MXS IC. The 9V Vbatt rail is pulling 18mA at idle now, which appears to be normal. I think the trick now is to check the UART debug communication and see if an error is being thrown out there.

[Towlerar]

new fotos of the MC9328 & the Falsh & RAM

May I ask where you sourced your RAM from? I cant find it stocked anywhere.

[Towlerar]

Today I attempted to get the debug data out of the UART port on the MXS IC. Try as I might I could not get the data out. My understanding is that the baud rate is 115200, 8, 1, N but I was not able to receive in TeraTerm or even decode with my Rigol DS2302, despite there clearly being data there. 3V TTL with nice, clean rise and fall. I was probably making some stupid mistake. Either way, the point is moot I guess because I made progress even without that data. I'll probably go back and figure out what I was doing wrong. I would expect even if my baud rate is wrong I would get gibberish.

Anyway, I took a break from the UART problem and decided to look at the weird issue where the LCD worked once and did not work again. I started by flowing contact cleaner into the flat flex connector and inserted/removed the flat flex several time even though both sides are gold plated and shouldn't oxidize. Then I heated up the solder joints on the connector pins, still no change to the display. Next, I pressed on the flat flex it multiple directions to see if it would come good and to my surprise I got a partial image. I then took the LCD out and inspected the flat flex under the microscope. The edge of the connector where the wipers contacted the gold pads looked fine. However, I noticed a micro strain hairline crack on the traces coming off the pads where the area of the flat flex with stiffener exited the connector body. It probably had small vibrations over long periods of time which ultimately strained the copper and cracked it. I'm guessing this meter had a hard life riding in a truck mounted toolbox based on the condition of the outside case. That can obviously set up sustained vibrations at certain vehicle speeds and eventually crack. Being that these LCDs sell for near $200USD on eBay, I was keen to try to fix it myself. I essentially  tinned the gold plated fingers with solder and cleaned it with alcohol. I know it's not the best for longevity,  but there weren't a lot of options. Also, this meter is now in a form of retirement and will have a cushy life on a bench in a temperature and humidity-controlled lab. The clearance became a bit tighter in the connector, but it worked! Again, I'm unsure what kind of longevity it will enjoy. I was still not getting the meter to fully boot, and given my UART problems I decided to try something different. I used some thin liquid flux and dropped it under the MXS applications processor and tried reflowing it at 315C with the hot air rework station. To my surprise this seems to gave worked! There was a definite improvement in how far the meter booted. However, at this point I was getting just a black line where the measurement should be regardless of what position the rotary switch was in. I verified that the notches between the switch and the housing lines up correctly. Then it struck me that if the switch was not making good contact with the PCB, measurements would not be able to come through. I cleaned the PCB and switch contacts with contact cleaner and rotated the switch back-and-forth several time and tested again. And... It's alive! It's almost a bit disappointing I didnt have to get into flashing any ICs. I think I'll still go back and figure out the UART issue that clearly is attributed to my setup a f not the meter itself. I'm also going to go forward with getting a JTAG programmer and will carefully probe around the MSP430 and MXS. It wouldn't hurt to have uploads of their hex files for the future. I'll probably also buy a spare MXS chip since those are EOL. I already cant find the RAM chips. The NOR flash and MSP are still available so I wont worry about that yet. I'll go ahead and order some replacement parts and get it put back together. This should just about wrap up troubleshooting for me. I just need to run some performance tests using my various calibrator to ensure everything meets spec. Who knows, perhaps there is still some calibration or even more troubleshooting in my future.

[giosif]

Hi,

Congrats on resurrecting your multimeter and I hope it serves you for a long time!
About the baud rate, I've used the info on the page at the below link to determine unknown rates:
https://www.kumari.net/index.php/random/37-determing-unknown-baud-rate

HTH

[MihaiV]

Hello,

Does the MSP430 check the analog 5V regulator and the voltage reference or just use it as external reference? My unit enables the 5v regulator but it shuts down immediately (photo in a previous post). I have not seen anything related to power coming from the analog side to any pin or other component near the MSP. My unit displays the start screen ("Fluke true rms....") but instead of measurements it displays a dark screen). Dark screen appears when the led turns off.

[Towlerar]

Hello,

Does the MSP430 check the analog 5V regulator and the voltage reference or just use it as external reference? My unit enables the 5v regulator but it shuts down immediately (photo in a previous post). I have not seen anything related to power coming from the analog side to any pin or other component near the MSP. My unit displays the start screen ("Fluke true rms....") but instead of measurements it displays a dark screen). Dark screen appears when the led turns off.

As far as I am aware, the MSP430 only checks the 2.5V reference voltage level. It's more likely that the MSP430 is shutting down the 5V rail as you stated. However, the 2.5V voltage reference IC that the MSP430 monitors (U42) is fed from the 5V regulator. So if there is no 5V, there's no reference voltage.

If everything seems normal with the regulators, I would also recommend checking the UART output from the MXS IC (U26). An easy point to tap into is the unpopulated J5 at the top of the PCB. Pin 3 is TX on U26, Pin 1 is RX on U26, Pin 2 is GND and the baud rate is 115200,8,1,N. This should give you some debug info and may give you insight as to why the unit is not completing booting. Also, if the rotary switch is dirty, not making good contact, or is in an invalid rotation, you also won't get any measurements on the screen after booting.

[MihaiV]

Hi Towlerar,

The 5V regulator is indeed disabled by the MSP. In a previous post there is a scope capture with it. I also checked the UART output and it shows a clean entry.

With a few precautions, I powered the 5V rail from a different source, after powering the rest of the meter, and there were no problems with the current consumption or voltage outputs. 5V was clean and so was the reference. A quick check with a thermal camera did not show any component warming up. But the behavior remained the same, meaning that it was not expecting something from the analog side.

Another thing I checked was the SPI bus between the MXS and MSP. I managed to see the MISO and MOSI signals. When powered and turned on, in the gray screen state, i can see some messages being sent by the MSP, most likely a shut-down command, but nothing happens. Perhaps that ball under the BGA (or even a via) is cracked. I will return with some scope prints with the messages.

By the way, I had to change the MXS crystal because there was no oscillation on one side of it. The chip was driving but the crystal was not resonating. After changing it with something from my junkbox is started oscillating strong and clean (see post 171 at page 7).

I assume that the crystal was broken due to a strong mechanical shock, so there may be a cracked ball somewhere. I already reflowed it once, but to no success. What else could i check?

[Towlerar]

Hi Towlerar,

The 5V regulator is indeed disabled by the MSP. In a previous post there is a scope capture with it. I also checked the UART output and it shows a clean entry.

With a few precautions, I powered the 5V rail from a different source, after powering the rest of the meter, and there were no problems with the current consumption or voltage outputs. 5V was clean and so was the reference. A quick check with a thermal camera did not show any component warming up. But the behavior remained the same, meaning that it was not expecting something from the analog side.

Another thing I checked was the SPI bus between the MXS and MSP. I managed to see the MISO and MOSI signals. When powered and turned on, in the gray screen state, i can see some messages being sent by the MSP, most likely a shut-down command, but nothing happens. Perhaps that ball under the BGA (or even a via) is cracked. I will return with some scope prints with the messages.

By the way, I had to change the MXS crystal because there was no oscillation on one side of it. The chip was driving but the crystal was not resonating. After changing it with something from my junkbox is started oscillating strong and clean (see post 171 at page 7).

I assume that the crystal was broken due to a strong mechanical shock, so there may be a cracked ball somewhere. I already reflowed it once, but to no success. What else could i check?

It seems to me that since you have a clean boot-up (based on your screen capture) that the issue is likely not with the MXS, MSP, or Flash. The problem could be with bad sectors in RAM - I'm not sure how involved RAM is in the regular operation of the meter vs. simply data logging.

My fear is that there is a problem with U9, the measurement processor, which is a Fluke branded item. I'm not sure whether or not that part is available. You could check the SPI comms between the MSP and U9. On the MSP430, the following pins are SPI to the measurement processor - 63 CS, 50 SDI, 48 SCK. There is also a reset pin which comes from pin 51 on the MSP.

Since the MSP is looking for a valid rotation of the rotary switch, and you've verified that it's in a valid rotation and clean, try checking the resistor ladder it uses for determining which position it is in. R30-R35, R40-44 are used as a large resistor divider, each going to a switch position to tell the MSP what position the switch is in. I would make sure all of those values are good.

[MihaiV]

Hello Towlerar,

Thank you for the advice.
I checked first that the contacts of the rotary switch are clean. Next, I checked that it was not installed in a wrong way. Someone was inside the meter before me (the heads of the input screws are slightly damaged). Since all parts are indexed, there is no way to assemble it wrong. I removed it to gain access to the contacts.

Next, i checked all vias around the switch, by buzzing to some visible traces through the PCB (all ok).

A visual inspection and a buzz of each branch of the resistor divider revealed the contacts at the switch. I traced a crude schematic (in the attachement). Pin 95 of the MSP appears to be connected to a position in the resistor ladder. It is P6.0/A0, obviously used as ADC input.

First, I must say that the 5V regulator is immediately disabled by the MSP, so I had to remove R88 (a 0R resistor) and to apply 5V externally. I noticed that if I turn the 5V too soon after pushing the power button the boot logo does not appear anymore and the power led remains on. If i apply 5v later (or never) it shows the boot logo, and the led turns off, simultaneously bringing the gray screen.

The 5V rail takes around 1mA (indicated by the power supply). I will measure it with a proper meter at the next debugging session.

Next, I measured all voltages of the resistor ladder with the rotary switch removed (see photos, positions on the ladder marked in red). They appear fairly well spaced apart in voltage, and are stable.

I buzzed the MSP pin, to see where it connects on different functions (with power removed, of course). The results are in the top right corner of the photo. Interestingly, some functions connect the pin at 2 positions in the ladder (VAC, mVAC, VDC). The others connect to a single position.

After that, I applied power an measured the voltages on each function. And they do not match the ones with the rotary switch removed! Obviously, the voltages in the ladder are also deranged. The values are indicated in the top right of the attached photo. I started buzzing all over the board, but couldn't find another connection anywhere else.

Is guess that is not normal. Is it?

It is a bit late here and my eyes are burning, so i will continue tomorrow, before i damage something else. I will try to hardwire one of the functions to the pin (like the VAC), without the rotary switch, and see what happens.

[Towlerar]

Hello Towlerar,

Thank you for the advice.
I checked first that the contacts of the rotary switch are clean. Next, I checked that it was not installed in a wrong way. Someone was inside the meter before me (the heads of the input screws are slightly damaged). Since all parts are indexed, there is no way to assemble it wrong. I removed it to gain access to the contacts.

Next, i checked all vias around the switch, by buzzing to some visible traces through the PCB (all ok).

A visual inspection and a buzz of each branch of the resistor divider revealed the contacts at the switch. I traced a crude schematic (in the attachement). Pin 95 of the MSP appears to be connected to a position in the resistor ladder. It is P6.0/A0, obviously used as ADC input.

First, I must say that the 5V regulator is immediately disabled by the MSP, so I had to remove R88 (a 0R resistor) and to apply 5V externally. I noticed that if I turn the 5V too soon after pushing the power button the boot logo does not appear anymore and the power led remains on. If i apply 5v later (or never) it shows the boot logo, and the led turns off, simultaneously bringing the gray screen.

The 5V rail takes around 1mA (indicated by the power supply). I will measure it with a proper meter at the next debugging session.

Next, I measured all voltages of the resistor ladder with the rotary switch removed (see photos, positions on the ladder marked in red). They appear fairly well spaced apart in voltage, and are stable.

I buzzed the MSP pin, to see where it connects on different functions (with power removed, of course). The results are in the top right corner of the photo. Interestingly, some functions connect the pin at 2 positions in the ladder (VAC, mVAC, VDC). The others connect to a single position.

After that, I applied power an measured the voltages on each function. And they do not match the ones with the rotary switch removed! Obviously, the voltages in the ladder are also deranged. The values are indicated in the top right of the attached photo. I started buzzing all over the board, but couldn't find another connection anywhere else.

Is guess that is not normal. Is it?

It is a bit late here and my eyes are burning, so i will continue tomorrow, before i damage something else. I will try to hardwire one of the functions to the pin (like the VAC), without the rotary switch, and see what happens.

Good work getting the rotary switch reverse engineered. It seems like you might possibly be onto something.

I just had another thought. Is it possible the MSP is disabling the 20V bias boost supply after bootup? The boost converter IC is U21 and the signal to enable it appears to be on the gate of Q43 PFET via a bootstrap transistor Q42.

[MihaiV]

Hello,

The 20V power supply is enabled at power-up and remains on as long as the meter is on. I can't turn it off without removing power. Holding the power button for more than 4 seconds reboots the unit. By the way, I have 21V, not 20. All other voltages generated by U19 are a bit higher too.

I tried to hard-wire one function to the MSP pin 95, but no results (the Ohms measurement). Tested with and without the rotary switch contacts, with and without external 5v supply.

But the fact that the voltages seen by the MSP are so different from the ones at the resistor ladder seems weird. I hope the selector was not changed with one of a different model, as someone was inside this unit before.

For a moment I suspected that the MSP pin 95 might be faulty and drag the voltages down, as they were connected to it by the selector, but hard-wiring proved it is ok, I measured 1.47V connected at position 5 (Ohms measurement).

Next, I will focus on the communication between the measurement processor and the MSP.

[MihaiV]

Hello everybody,

I tried some SPI bus probing between the MSP and the other components.
I started with the MSP to MXS communication, and I saw that the MSP sends commands and the MXS responds. I did not decode the actual commands, I wanted to see that the traces/vias are intact and that there is no short-circuit between them.

Waveforms in photo 1 appear when the power button is pressed first time after applying 9v. The timebase is large, so the clock appears simply as a few pixels (channel 1 is CS, channel 2 is the CLK, channel 3 is MISO and  4 is MOSI).

Photo 2 is zoomed in a little; it seems that there is some communication with the CS line high too and the messages are different (photos 3 and 4). No activity appears on the bus when other buttons are pushed or the rotary switch is switched (not even with 5v applied externaly). But the traces up to the MXS are ok.

Next, I tried the MSX to ADG714. Accodrind to post 153 from KJK24, MSP pin 48 is clk, 50 SDI and 49 SDO. There is no activity here at all, with or without 5V from outside. A quick buzz indicated that pins 48 and 50 buzz to pins 1 and 3 (SCLK and DIN) respectively on the ADG714 switches. Pin 48 doesn't ring to pin 22 (DOUT) of the ADG! It goes to R291 (unpopulated) and nowhere else on the board. Pin 22 of ADG does not buzz anywhere, too.

I checked the photo from post 153 because I remembered it had a photo of the vias under the MSP but noticed that my board has a slightly different layout. My DIN trace goes away from the MSP, to the south-west part of the board (photo 5), but the photo in 153 goes right under the IC. Seeing post 139 I noticed that I have a ton of parts unpopulated in the south of MSP (my unit is 287/NUC). My board says FLUKE-28x-3001 REV 015.

Can anyone check that pin 49 of MSP is in direct connection to pin 22 on the ADG714 on the other side of the board (U48) ?

Thank you in advance!

[MihaiV]

Hi,

Ignore the second part of my previous message... Just being dumb 
I rechecked the datasheet and DOUT of ADG714 does not need to connect to anything (unless you wnat do daisy-chain multiple devices). No need for it to go anywhere.
Took a photo of what's under the ADG, while at it.
Back to square 1. And to the SPI between MXS and the measurement processor.

[MihaiV]

Hello again,

First with the (somewhat) good news: the LTC2415-1 (U10) ADC appears to be working. Attachment 1 shows the CS, SCK and SDO signals. The CS doesn't look too good with that falling edge. The data and the clock look good though.

The ADC is powered by the 5V rail, so the signals will have the same level. On the board, the CS and the SCK are connected directly to the MSP, but the SDO connects through a large resistor (measured ~50k in circuit). It seems that the ADC is operating alone right now, meaning that SCK is an output. AS the datasheet says, if SCK is HIGH or floating at power-up or during a High-to-Low transition, the converter enters SCK internal mode. Might be the reason for the dodgy CS signal too.

I believe it operates with SCK from the MSP in normal mode, because if I power it up with the scope probe attached to SCK it does not output anything, but if i power it without and attach it later it outputs clock and data.

Photos 2 and 3 show a weird data output (blue trace mostly High), and a zoom of the falling edge of CS.

And here is the bad news.
I just checked the SPI connection to U9 (SL10327) and nothing happens there, with or without external 5V. Checked absolutely every pin, powered up the meter on each one and I only see something on the power and the reference voltage.

Good news is that I found the IC on ebay.

Could anybody post a scope capture of what happens on the SPI bus of a functional unit ? Please?

Thanks,
MihaiV.

[MihaiV]

Anyone?

[booyeah]

From a good 289:

1. U10 (LTC2415), attachment 1 and 2 -  CS bar, SCK, SDO.    CS bar is directly connected to MSP 47 , but couldn't find where SCK and SDO went.

2. U9 (SL10327), attachment 3, 4, 5 -   CS*(26) to msp (63), SCK (28) to msp (48), SDI (27) to msp (50).
 
Probe connections aren't the best, so some extra ringing.

[MihaiV]

Hello Booyeah,

Thank you for taking the time to open your meter and investigate the signals!
My U9 appears very dead, so I ordered one from ebay(most likely removed from some other meter). There are some chances to be faulty too, but I'll take my chances for 5$.

Thanks,
MihaiV.

[MihaiV]

Hello again,

The SL10327 arrived! The part appears new, in the tape, with no traces of removal from other equipment.
Swap it in and... nothing.
The problem is somewhere else.
Don't know where else to look at the moment...
Could it be a problem from the MXS or the RAM? Could the start-up logo be displayed from the ROM and when RAM operations start, everything goes wrong?

[booyeah]

I'd like to say a massive thanks to everyone in this thread.

A couple of years ago I decided to make a lipo battery circuit for my 287, I either reversed the polarity or possibly over volted the battery inputs. Anyway the end result was a dead meter with just the five power button flashes. I had a go at troubleshooting but got nowhere, so it's just been sitting in a drawer ever since.

After following this thread, I decided to have another go at fixing it.
There was a short on the MXS 3v rail. Testing with an external supply caused the CPU to heat up. So I ordered a replacement and had a go at soldering it with a cheap ebay hot air gun.

I couldn't do it as the lead free solder paste just wouldn't melt properly. I hand re-balled the chip with some leaded solder and thankfully it worked this time. I then edited the MXS boot script using an Arduino as a serial pass through. Success, my 287 has come back to life.

Something to note is how delicate/finicky the lcd flex can be.
Initially I thought I had damaged the lcd as well, as it was very faint on booting. The fluke logo would show faintly them fade to nothing. I tried it on another meter with similar results, sometimes there were graphical lines missing as well.

I checked all of the connector pin voltages and signals, they roughly agreed between the two meters. I tried re-seating the connector, cleaning etc. I found that applying a little bit of pressure to one side of the flex connector fixed it, so I tinned the pads a little bit and now the screen works perfect.

[booyeah]

Hello again,

The SL10327 arrived! The part appears new, in the tape, with no traces of removal from other equipment.
Swap it in and... nothing.
The problem is somewhere else.
Don't know where else to look at the moment...
Could it be a problem from the MXS or the RAM? Could the start-up logo be displayed from the ROM and when RAM operations start, everything goes wrong?

In an earlier post you checked the lcd pins. Is it possible the solder connections to the cpu are damaged / cracked?

My    287,     289   lcd connector
1  20.349   21.19
2  19.017   19.88
3  17.83     18.56
4   2.525     2.637
5   1.272     1.32
6   0            0
7   0            0
8   2.953     2.931
9    pic 1
10  pic 2
11  pic 3
12  2.953    2.934
13  pic 4
14  pic 5
15  pic 6
16  very busy like pic 6
17  very busy like pic 6
18  0          0
19  6.986    6.98
20  4.758    4.69

Pins 13-17 were very busy non repetitive signals so hard to capture, but you should be seeing some activity, not 0 volts.

[MihaiV]

Good evening booyeah,

Thank you for the scope captures! There is a ton of signal missing from my LCD lines!
I will check them again in case I missed them somehow.
I took pins 19 and 20 as backlight circuitry, with Q17, Q18 and Q18 part of the intensity adjustment. There is nothing there too.
I will try to reverse engineer that part and come with a schematic.
Was your backlight on?

I also fear(ed) that there is a solder joint cracked somewhere. In fact, the meter was rarely showing the start logo. I noticed that it worked constantly only when pressing near or on the MXS. I reflowed it once with some equipment at university and it booted every time since then. Also reflowed RAM and Flash while at it.

Also, the MXS crystal was not oscillating at all (but it was still showing the start logo). I changed it with something from the junkbox, and now it oscillates nicely (pics in a previous post). Changed nothing though.

Broken joints and a possibly cracked crystal could indicate a healthy mechanical schock (although the case is like new).
One thing I could try is to reflow the MXS again and gently move the part while in liquidus phase (with a bucket of flux). Maybe the cracked joints are oxidated and did not allow the solder ball to regain continuity. I have seen that before in other equipment.
Will return with updated status on those LCD lines.

Thank you,
Vidrascu Mihai.

[MihaiV]

Hello,

Back with the LCD pins scope captures.
My final pins are definitely silent (see attached pictures).
I used the low(er) capacity probe at work to also check the crystals of the MSP and MXS, also in the attachments.
I did not manage to reflow the MXS yet, no spare time at work the last days.
Anyway, isn't the LCD communication unidirectional? The meter should work (beep at button push or at continuity test) even if the screen is detached (or with a faulty solder joint).

Happy Holidays!
MihaiV.

[MihaiV]

The remaining scope captures, attached below.