Fraser has two new patients inbound - E2V ARGUS 4 & Seek Reveal Pro FF

[Fraser]

Just one more comment. If you look at LDO Output image 6 and check the period of the signal 'block' repetition you will find it to be around 62.6ms. That is around 16Hz  We can guess that the signal is related to ROIC frame rate output (this is a >15fps PRO core). All food for thought.

I have a feeling that this fault is not going to be as simple as I first thought  All good fun though 

Fraser

[IwuzBornanerd]

I "wuzgonna" make similar comment.  Now you know what that ">15Hz" means. 

My question is what is going on during that period of time when there is essentially no noise?  If you take just the period of time where all the noise is and eliminate the gap you would get the 18fps that I get with the USB comm in a separate thread.  So I wonder if all the noise is occurring during data transfer only & the processing time for the previous frame is the whole 62.5ms.  But if there is no USB interface in that thing, why would the data transfer take so long?

Watching you say it was time for bed & then half an hour later making another post tells me what I would be like if I had a stupidphone...Oh, while it's on, one more thing... 

[Fraser]

OK readers, just like a Dick Tracey story.... we return for another thrill filled episode in the voyage of discovery that is the Seek Reveal Pro with noisy imaging 

(For youngsters..... https://en.wikipedia.org/wiki/Dick_Tracy  )

When I left you last, I had completed making decent DSO captures of the input and output of the LDO regulator that is positioned on the microbolometer PCB. As I previously stated in this thread, any noise on the supply rail to the analogue parts of the microbolometer/ROIC combination can lead to noise injection on the output image data. The question has to be, how much noise does it take to be a problem to a microbolometer and ROIC. The answer is... I do not know, but the less the better in my book. At this point I refer the reader to the previously posted specifications of the TI LDO regulator that I believe to be the one used on the Microbolometer core. The specified noise output is very small indeed and what I would expect from an Ultra Low noise LDO circuit. If you have read the article on LDO behaviour with noise bypass capacitors, you will also know that there is the risk of a noise prone output if the values of the bypass capacitors are not chosen with some thought. Failure of bypass capacitors can cause significant issues as the LDO is not designed to operate without their capacitance present at its I/O. The level of noise that I saw at the output of the LDO on the microbolometer PCB was surprising and a little puzzling at the same time. The Pk-Pk levels were up to around 100mV but the noise was not random oscillations of an unstable LDO, it was structured and it very quickly became apparent that the noise was a crib of the ROIC's activities with complete frames clearly discernable and the frame rate easily measured at 16fps. Hmmmm that was not expected and made me a little concerned. Such a signal would normally be heavily suppressed as modulating a power rail is a tad naughty. However, it is possible to modulate a power rail through inherent ac loading if the power rail is already in distress and has high impedance or poor bypassing. This I suspect could be our issue, rather than the less appealing thought of a failure in the ROIC structure. The fact that the core produces images suggest that the ROIC is still operating pretty well, albeit with noise bars.

Today I cheated ! 

Unlike many who might be attempting this sort of investigation, I am fortunate enough to have two other Seek Reveal PRO cameras. One is standard <9fps whilst the other is Fast Frame spec.

Readers may recall that I previously stated that I purchased this sick Reveal Pro FF in order to learn about its design and behaviour. I did not fancy messing around with my brand new Reveal Pro FF as it remains mint    Now a faulty unit at reasonable cost, that is something to play with 

Now that I had my DSO captures from the sick Reveal Pro, I thought it would be both interesting and helpful to take the same measurements on a healthy Reveal Pro. I could then compare the noise levels at the LDO to determine what was 'normal' in that region of the design. Now I did not want to mess around with my new Pro FF so I used the Standard <9fps Reveal pro instead. They should be very similar, or even the same after all ..... or are they ? 

Access to the Core on a Reveal camera is so simple. You just remove the two screws that retain the metal front casing section. The core may then be easily accessed or removed 

Opening another camera that is known to be healthy presents another opportunity, but one that must be used with great care ! It would be possible to test the suspect core on the known good camera mainboard and test the known good core on the sick cameras mainboard. This type of parts swapping is not without risk ! This is especially so if the two cameras are not identical hardware and firmware. Firmware can throw a wobbly or even be corrupted by parts swapping without thought. In my case I was concerned about the possibility of a <9fps camera mainboard somehow setting a frame rate restriction on a Fast Frame capable core. It was a risk and one I had to accept if I were to do the swap.

The first set of pictures that I am going to upload here are of that core swap and the images produced. I will detail the DSO test results afterwards 

Fraser   

[Fraser]

First the Suspect core connected to the known good mainboard. I fully expected a poor picture and potentially no picture as I am using a FF core on a non FF camera chassis !

The results are below..... look for the tell tale noise bars and ignore the actual scene detail ! 

[Fraser]

Now the known good core attached to the sick cameras mainboard. Again, I was not expecting perfect images and potentially, no image at all.

What I got was unexpected though ! The core initially looked like it would not produce an image...... but then as time moved on and multiple FFC events occurred, I saw a strange effect.  I began to see a negative image. That is to say, although set to White Hot, I was seeing a hot coffee cup as black hot. then I saw why, the image was not updating until an FFC event. Wierd but I believe this is connected with how the FFC was operating on this <9fps core. More on that core in a minute 

To the pictures...... they were taken over a short period of time and the initial poor picture was replaced with the unusual inverted still images ! It looks like FFC is operating in reverse.

Please note that initially the camera was looking at a pretty low Delta T scene of a cabinet against a wall. It was after I discovered the inverse image that I used the warm coffee cup as a target and set the colour palette to Iron.

[Fraser]

So waht about the frame rate restricted core ? Anything different about it physically ?

Well yes, it is a different design of PCB to that in the sick Reveal Pro FF. Take a look at the attached pictures. The sick Reveal Pro FF core is already known to us from my previous pictures. The Standard frame rate limited Reveal Pro core is the type seen in Mike's teardown video and is equipped with an M0 processor ! The good news is that the LDO regulator is basically the same so any comparison tests for noise levels on the output should remain reasonably valid with the potential for more rail noise on the frame rate restricted core due to the presence of the M0 processor.

I attach the comparison pictures of the two cores. The blue PCB is the Pro FF and the Green PCB is the Frame rate limited Pro core.

[Fraser]

More pictures of the frame rate limited Pro core.......

[Fraser]

OK, time for me to take a short break from the computer.

So what did we discover from the core swap test ?

Well I confirmed that the sick Reveal Pro mainboard is working without obvious issues. The image inversion is likely to be an incompatibility issue as I was using a standard core on an FF mainboard.

The Fast Frame Core from the sick Reveal Pro displayed evidence of the noise banding when connected to the known good Reveal Pro mainboard. The core is where the problem lies and it is not some weird mainboard issue or mainboard supply circuit noise levels.

So what will the DSO tell us about the noise levels that should be present on the power rails of a working example of the Reveal Pro camera core ?

That comes in the next ''episode'' 

Come back later for more ...........

[Fraser]

We return to the story dear readers........

In the last episode we saw that the restricted frame rate Reveal Pro uses a different core PCB to that of the Reveal Pro Fast Frame model that I am working on. This could be just a revision related matter but, as Mike said in his teardown.... what is that M0 processor doing ?

We also saw that the Core from a Fast Frame Reveal Pro cannot be just swapped with that of a frame rate restricted model.

Most importantly for this investigation, we saw that the sick Reveal Pro FF core is most definitely the cause of the noise bars that are displayed on its host cameras screen. It is always nice to prove the location of a fault to avoid chasing a red herring !

So what comes next ? Well, as I said, I repeated my DSO tests on the LDO regulator of the known good cameras microbolometer PCB. Despite the differences between the FF core and standard core, the noise level should be comparable in level. What follows are comparisons between the sick core inputs and outputs with those of the known good core. The image captures are named and the good core is ''StdCxxxxxx".

Fraser

[Fraser]

LDO Regulator input Noise level comparisons

Sick core first, then known good core.......
 

[Fraser]

Yes, surprisingly the healthy camera has higher noise levels at the input to its cores LDO regulator.

So what about the levels on the output side of the LDO regulator ? I can say that it was a surprise..... I am off to enjoy a coffee and some cake so will upload the output waveforms soon 

Watch this space.....

Fraser

[Fraser]

OK, drum roll please........ here are the comparison DSO captures of the LDO regulator output on each core. This is the important one. It defines what is 'normal' by using a known good core as a reference.

As in the previous post, I will show the sick cores captures first, then the healthy cores captures for comparison........

Fraser

[Fraser]

So is this test a joke or in some way wrong ? .... Nope, I was so surprised with what I saw that I tested, tested and tested again to confirm the readings. There was so little noise on the LDO output of the healthy core that I had to check the layout of the PCB around the LDO to ensure I was checking the LDO output pin.

I think we can safely say that the noise levels present on the sick cores LDO output are not normal and causing the noise bars. So what is causing this high level of noise ? My money is on a failed LDO output bypass capacitor (cracked?)

More from me when I have replaced the capacitors as that is the next logical move. If that does not cure the issue, I will need to fit a new regulator as the next step. After that I am faced with a failed ROIC which I hope it is not.

Fraser

[Fraser]

For anyone waiting to see the hidden side of the Reveal PRO FF core Microbolometer PCB, here it is 

I removed the PCB to inspect the Microbolometer bond wires  As can be seen, not much to see and definitely no Cortex M0 processor hiding here.

Fraser

[Fraser]

Now the Core chassis rear detail.......

[Fraser]

Bond wire images. I checked them with a microscope but my camera just about sees enough detail. No issues were found with the bond wires.

Why did I check them ? Simple..... exclude all reasonable possibilities before reaching for the soldering iron. There was the possibility of shorting adjacent bond wires or contamination/corrosion between them.
It only takes a minute to check and is far better than feeling silly later for not doing so and finding an obvious issue  Been there, done that, got the T shirt !

Fraser

[Fraser]

Second bond wire picture....

[Fraser]

Now by this time I feel sure some readers will be thinking “why doesn’t he just fit a new pair of MLC capacitors and have done with this repair”. Well investigating the fault in this particular camera was not so much about Repairing it as quickly and efficiently as possible, but more of a journey into camera design and the collection of some useful information in the process of identifying the failed component. I hope my commentary also provides some insight into how I approach repairs on cameras for which I have no service information or schematics. Diving straight in and replacing any parts that ‘may’ be the cause of the problem can be a mistake, especially if you accidentally introduce additional faults in the process. Fitting components that are not the same spec as the original parts can also ‘muddy the waters’ if the system does not get on with their specs. By studying the ‘patient’ to a reasonable depth, you can often determine its needs where component specifications are being considered. This particular camera has a reasonable straight forward fault that I could likely diagnose, and determine whether it can be repaired, in less than an hour. I might be able to repair it in that time as well. But then where is the fun and learning in that eh? Sadly I also have very little time to spend on ‘toys’ so this investigation has had to fit into small time slots that are not ideal for reverse engineering PCB’s.

All good fun for me though so I hope this ‘Saga’ is enjoyable to some readers 

Fraser

[IwuzBornanerd]

If you figure 2.8V as full scale on the 14 bit ADC, then the LSB represents 0.17mV, and even the amount of noise you see on the output of the good unit is too much noise. 

[Fraser]

Room for improvement possibly ?   

The LDO regulator noise spec is excellent so it is what has been done/not done on the a Microbolometer PCB that influences the higher noise levels present. A lack of suitably positioned bypass capacitors is a possibility, along with PCB tracks acting as inductors and excessive noise levels on the input side of the LDO. This could be an interesting little project for the future.

Fraser

[Fraser]

Just ordered some LP5907 2.8V LDO regulators to play with. They are only 40p each and I still suspect I may need to fit one to the sick Seek Reveal Pro.

I have decided to fit 2.2uF 10V X5R MLCC’s in place of the input and output bypass capacitors currently installed. With voltage derating considered they still retain plenty of capacity for their role. I will be adding an extra MLCC to the input side of the LDO to see if it lowers the noise at that point. I may add additional MLCC’s on the supply rail served by the LDO to see if noise from the Microbolometer ROIC can be further reduced.

Fraser

[Fraser]

For those readers who have stuck with this thread so far...... I salute your staying power 

Tonight was the night of the soldering iron and it had work to do ! I was going to remove the suspect MLCC capacitor on the output of the LDO regulator and fit a nice shiny new Wurth 2.2uF 10V X5R part.

This was only ever going to be a quick test as I was expecting disappointment and more work will be needed on this core.

So the deed was done and the new MLCC was fitted. Those 0402 capacitors are little blighters.... if they ping out of the tweezers, they are gone for good  The Gods were on my side today though and I got the little blighter soldered to the PCB on the first attempt. I would normally use a HAP (Hot Air Pencil) but the ground plane on this Microbolometer PCB is formidable ! The new MLCC is only fitted for this test so please do not look too closely at the soldering. It was about speed rather than a pretty fillet of Tin&Lead ! The Ground plane side of the MLCC was as challenging to solder as I expected. I had broken out a new sharp tip soldering cartridge for this task and it took a while to get the pad up to temperature. All came good in the end though and a usable pair of solder joints resulted 

Well did it make any difference to the fault symptoms ? I connected up the core to its host PCB and switched it on..... and ......... see the pictures for more 

Fraser 

[Fraser]

First Power on test after the new MLCC was fitted.......

[Fraser]

The ubiquitous Coffee Cup test 

[Fraser]

As can be seen from the pictures.... normal service has been restored 

There is no interference present on the display and the scene images are great. The temperature measurement is now accurate 

This camera had suffered the failure of one tiny 30 Cent component. I have that component safe and will test it. I suspect it has suffered a stress fracture and lost capacity.

Well we got there in the end and the journey was fun for me. I have not finished yet though as I will properly rework the LDO section of the Microbolometer PCB and fit new 2.2uF MLCC capacitors on both the input and output sided of the regulator. I will then use the DSO to capture the noise levels at the input and output of the LDO for my record and to aid others on the forum.

I have been reverse engineering the Microbolometer input to the mainboard and some other parts of it. That is a work in progress and is time consuming. A story for another day and another post me thinks.

I will upload the pictures I captured on the Reveal Pro camera after repair in a minute for readers interest.

The Argus 4 awaits my attention next in the queue 

Fraser