Consultant Recommendation Request

[Poe]

We're looking for a troubleshooting consultant in the USA, preferably the East coast.

We need to identify the root cause of failure for a new batch of boards since we lack the equipment and expertise.

The boards have the following elements:

a high speed USB to Ethernet controller hub
a USB 3.0 hub
a TMDS to tablet interface for an HDMI port
a couple SMPS and power management circuitry
a RS232 serial port
No firmware

Background:
Ten years ago our company paid to have a board designed.  A Chinese CM has been building it ever since and they all still pass test.
The original designer made at least six revision changes over those ten years.  There could have been more, but our documentation only goes to Rev7.

Now the design company is gone and the Chinese CM is not cooperating. 

We tried to build a batch of boards with a different CM, but they're all failing with the same symptom.

It's possible that the Chinese CM and/or the designer could have been asked to make changes that we never documented.
Although the new CM might also have build issues.  This is what we need someone to help us determine.

Any consultant suggestions are appreciated.  Thank you.

[nctnico]

What is wrong with sending a working and non-working board to someone for investigation? Do you really need someone close to 'home'? (or even on-site?)

[2N3055]

I agree with Nico, it is too vague a description.
You might as well send it anywhere, sign a contract or NDA with consultant and that's it..
What is the fault? Do you need USB troubleshooting, HDMI or power supply? Or you have EMC related problem so you need EMC expert.

From what you're saying it might as well be that manufacturers "upgraded" some of the components  to "new upgraded process", so they have same nominal function but different electrical specs.

More details please.

[asmi]

When I hear of a sudden weird-ass behavior of a high-speed PCB, the first thing that comes to my mind is "the stackup" (both composition and physical structure).
But then again, we don't really know what exactly "failing" means, and what symptoms are.

[T3sl4co1l]

I could take a look at it.  There may be value had with a schematic review, PCB review (if the EDA files aren't in a compatible format, then PDFs and gerbers at least can be studied), or about the fabrication/assembly package.  I guess this would incur somewhere between hours and days of work, depending on level of detail desired, and how much supporting information is available.  (I'm not east coast, so it probably wouldn't be practical to study in person, but it would be worth mailing samples, for instance.)

Hmm, USB3?  Ten years old?  *Checks* dang, that does check out... just barely!  That must've been pretty cutting edge back in the day.

Tim

[AndyC_772]

It's very much the sort of thing I could do too, though I'm probably a bit far away in this case.

The alarm bell for me here is "no firmware", yet there's a few devices on that board that must surely require configuration if nothing else.

Does "no firmware" really mean "none of the parts on this board require any firmware", or "we've never been given the firmware that these boards require in order to function"?

[Ice-Tea]

Poe, in case you don't know Tim: if he's willing to take a look at your problem you should be as happy as a clam and jump at the opportunity.

[2N3055]

It's very much the sort of thing I could do too, though I'm probably a bit far away in this case.

The alarm bell for me here is "no firmware", yet there's a few devices on that board that must surely require configuration if nothing else.

Does "no firmware" really mean "none of the parts on this board require any firmware", or "we've never been given the firmware that these boards require in order to function"?

Exactly that. Even if there is no customer processor/FPGA/CPLD on board, USB3 controller certainly has firmware, if nothing else internal one with manufacturer, and it is million miles today from what it was when thing was designed. It might be problem with that, for instance.

Regards,

[2N3055]

Hmm, USB3?  Ten years old?  *Checks* dang, that does check out... just barely!  That must've been pretty cutting edge back in the day.

Tim

That, or we're getting old.. dang...
 
Regards,
Sinisa

[jonpaul]

Hello Mr POE: If you have a serious issue that involves production boards, forums may not solve. A profession approach is more effective.

These are complex problems, requiring a deep study of the boards, docs and system, so an experienced expert on high speed boards, manufacture and assembly should be used.

For instance Dr Howard JOHNSON. http://www.sigcon.com/

 Expect to pay $300-600 per hour and its possible that the entire system or boards and all docs must be release to the consultant.

A contract and purchase order can precise all the details and responsibilities.

You can start by writing an RFQ, specifying exactly the entire system and issues.

Finally the entire affair highlight the many risks of outsourcing critical system components to Chinese assembly houses.

Bon Chance,

Jon

[Ice-Tea]

I'm not entirely convinced you need a 500$ consultant to *start* figuring this out. You need a decent engineer that can figure out if FW is loaded/needs to be loaded, check power supplies, check communication, whatever. Perhaps somewhere down to the road  you may need an expert, but not for the direction finding.

In addition, it has nothing to do with outsourcing to chinese manufacturers. It has to do with bad documentation and specification.

[AndyC_772]

Hello Mr POE: If you have a serious issue that involves production boards, forums may not solve. A profession approach is more effective.

The two are not mutually exclusive!

In any case, hiring a competent engineer with all the necessary skills and experience does not, I guarantee, cost anywhere near $300-$600/hr.

[jonpaul]

Hi: An normal engineer wont untangle this.

What do you pay for a good lawyer or CPA hourly rate to untangle legal or tax problems?


A consultant is different, normally a specialist.


Just my experience!

Bon Chance,

Jon

[nctnico]

Hello Mr POE: If you have a serious issue that involves production boards, forums may not solve. A profession approach is more effective.

The two are not mutually exclusive!

In any case, hiring a competent engineer with all the necessary skills and experience does not, I guarantee, cost anywhere near $300-$600/hr.

I agree. An engineer with experience in designing & troubleshooting high speed designs should be able to solve the problem in a couple of days at most and it is not going to cost anywhere near that amount per hour (I wish though... could retire early if people would pay me that).

[Poe]

Sorry it took so long to reply.

I'm in a completely different department/state than the team responsible for this assembly.  This was just dumped in my lap when the last engineer quit.  Management said there was no programming required, but that very well might be the problem.  My location just handles documentation so we don't have equipment to troubleshoot.

We have no problem sending the boards and test fixture out to be analyzed.  There are about one thousand assemblies failing test, so we can spare a few.

The test isn't very descriptive when something fails and I haven't seen it yet so I can't be more descriptive right now unfortunately.  Hopefully I'll get one in front of me soon.

About 20% of the BOM's passives are obsolete.  The new CM requested subs which appeared equivalent so we gave the green light.

Thank you all very much for the recommendations and for the offers.  We're going to have a meeting Monday with the new CM.  They or us will likely reach out afterwards.

[thermistor-guy]

...
It's possible that the Chinese CM and/or the designer could have been asked to make changes that we never documented.
Although the new CM might also have build issues.  This is what we need someone to help us determine.
...

Yes, I've seen this in practice, although some time ago.

In one case, the PCB files at the CM didn't match the schematic and PCB files at the design house. The designer made
some "last-minute" changes shortly before he left the company. It was a 5-person start-up, so forget about version control.
It wasn't just connectivity (tracks), there were also problems with PCB hole sizes for through-hole components.

Another case: a problem with PCB footprints. The PCB footprint for a high pin count, fine-pitch component was slightly off
 - the pitch was too small. The CM could sometimes make the component connect to the pads, just barely, with careful manual touch-up,
but not always. The CM informed the client company there was a problem, but somehow the message didn't sink in. You could just detect
the pin pitch discrepancy with the naked eye. Several revisions of the PCB went through the CM, with the same bad footprint.

Until one day, a particularly handsome and dashing engineer (ahem), new to the company, took it upon himself
to audit the board's PCB library, and clean up all the problems in one go. Rumor has it that he particularly enjoyed standing over
some snooty PCB layout specialists, demanding that they justify every dimension of every footprint ... NOW.

Auditing the design and build files you have, against what the old CM was supplying you, is a good place to start.

[Tomorokoshi]

Can you add a little more about this:

...
Now the design company is gone and the Chinese CM is not cooperating. 

We tried to build a batch of boards with a different CM, but they're all failing with the same symptom.

It's possible that the Chinese CM and/or the designer could have been asked to make changes that we never documented.
Although the new CM might also have build issues.  This is what we need someone to help us determine.
...

1. What was the relationship between the original design company and the original Chinese CM? Was the original Chinese CM subcontracted through the original design company? What made them become "not cooperating"? Is the problem one of not building the boards, not providing build documentation, not providing design files, etc.? Is there a translation problem? Do you have a representative in China?

2. Did the original Chinese CM order or manufacture the blank PCBs? Is the new CM using a different supplier? Note that sometimes either a CM or the PCB manufacturer will analyze the PCB Gerber files and determine that some dimension is not right, add some feature for the carrier pallets, etc. That means the actual delivered PCB may have some variation that isn't in your design files. Additionally, the PCB layer stackup thicknesses may not be the same. Check if those are specified on the PCB print.

3. Who managed the relationship with the original designer?

4. Whatever consulting or engineering route you go, make sure they get examples of the older functional boards. Allow enough that a couple can be modified to compare sub-circuits between the two builds.

[Ice-Tea]

Sending out a failed board is good, sending out a failed board together with a working board from back when "things were fine" is probably even better

[Poe]

Blueskull - Unfortunately all I have is a multimeter and a soldering iron from the early 80s. I'll see if I can check some of that though.  Thank you.

Thermistor-guy - I've experienced a scary number of big companies dependent on suppliers to handle their documentation control.  It's crazy.

Tomorokoshi - There was no relationship that I'm aware of between the company contracted to design and the original CM selected to build.  The new CM is using a different bare PCB supplier since we have no record of the original CM's bare PCB supplier. Our files appear to match the original CM's boards, but we can only do simple visual inspections.  We have a stackup which defines an impedance for each layer and thickness, but we can't tell if the new CM followed it.

[AndyC_772]

This really doesn't sound like a difficult problem; there's a hundred reasons why a new batch of boards from a new supplier may not work, and very few of them are as obscure or difficult to track down as passives that vary slightly between manufacturers.

Start with the basics. Look up all the ICs on the board, check their data sheets and see if any of them contain non-volatile memory. If they do, swap them between a working board and a non working one.

Check the supply rails. Check the clocks. Are they all ticking? If not, does something need programming? Some crystal oscillators come with factory programmed PLLs; do you have any of these, and are they programmed as they should be?

Test and see if anything on the board is working.

Check connectivity between the ports and the devices they connect to; it could just be a poor quality connector, or one which superficially resembles the correct type but has a different pin-out. (I've seen this, especially Ethernet).

Plenty of components come in subtly different variants - different speed grades, or versions with/without features enabled. Do the markings on the parts on the new boards match those on the working ones?

Are any parts held in reset? What normally causes them to come out of reset? Is there one part that wakes up the others?