How to test salvageable Xilinx UltraScale+ board from Ebay?

[lukego]

I'm a total newbie to electronics and I've made the rash decision to order a few boards from Ebay that include supposed-to-be-working Xilinx XCKU15P FPGAs. This is a $3500 part being sold for less than 1/10th of list price. I couldn't resist this because 100G ethernet is an important part of my domain of interest and this FPGA seems to support up to 8x100GbE links.

Here's the product on Ebay: https://www.ebay.com/itm/Xilinx-Kintex-XCKU15P-Fpga-On-Board/274145389372

The cards will arrive soon and I'm now wondering how to test that the FPGAs work (and otherwise return them.) Any ideas? How do you usually make sure the salvage parts you buy from Ebay are in working order? I'm planning to desolder these and put them onto custom boards but that will be well after the 30-day returns window and presumably desoldering them would void my quasi-warranty anyway :-)

These boards are supposed to be (damaged in some unspecified way) Mellanox Innova-2 Flex MNV303212A-ADLT cards. If I'm lucky I will be able to put them into PCIe slots on a Linux machine and probe the FPGA indirectly using Mellanox software. However I don't know if the board and/or ConnectX chips are in working order and so maybe I will need a more direct approach. I don't currently have Vivado or a JTAG programmer but I should be able to arrange access to these if needed.

Relatedly, what would be some good questions to ask the seller regarding testing?

Thanks in advance for any sage advice for this rash newbie...

[Haenk]

A proper cable does not cost a lot - so if you install two cards in two computers, you should have a working connection - if the FPGA is OK. (Note that the FPGA probably does come with a huge heatsink, so you would need appropriate cooling while testing.)
Another thing to note is, that those 100GbE-Adapters seem to have dropped in price a lot, there are plenty of those cards for far less than 350 USD around, however with the heatsink still installed, so one cannot be sure if this XILINX is on board (need to research that...)
 

[lukego]

A proper cable does not cost a lot - so if you install two cards in two computers, you should have a working connection - if the FPGA is OK.

Do you mean a cable for 100G ethernet? I do have that, and some other 25G/100G ethernet adapters, so I could test that way. But this would seem to depend on the board as a whole working which is in doubt.

Another thing to note is, that those 100GbE-Adapters seem to have dropped in price a lot, there are plenty of those cards for far less than 350 USD around

I should clarify: On the one hand it's true that 100G ethernet adapters are a relatively cheap commodity these days I don't think the same is true for FPGAs with SERDES capable of 25G/100G ethernet. It's the 25G/100G FPGA development platform that I want. So in a sense that I'm doing here is buying 32 x 32.75Gbps UltraScale+ GTY SERDES transceivers (for a song if they should happen to really work.)

[brabus]

(…) I'm planning to desolder these and put them onto custom boards (…)

 

Don't want to be rude, but those beasts are an authentic nightmare to solder even with specific tools (vapour phase soldering is a must). Any "home made" trial WILL end up damaging the FPGA and wasting 350$.
I would skip this phase and work on the development board until the VHDL script is satisfyingly robust.

Once the prototype works on the DevBoard, you can design the application PCB and have a brand new FPGA soldered through an EMS service. This is pretty much the only way to have it working at the end of the process.

If the seller says "used", it means that the boards still work, otherwise they shall be listed as "for parts/not working".

[lukego]

Any "home made" trial WILL end up damaging the FPGA and wasting 350$.

I'm planning to either resolder these FPGAs using hot air, or otherwise to dead bug them. I'll make a video to share my misadventures here so that others can learn from my folly

Maybe I'll start carving notches on my hot air gun handle for each thousand dollar chip I bugger up while trying to imitate Louis Rossmann...

[Fred27]

I'll make a video to share my misadventures here so that others can learn from my folly

If you're an electronics newbie than I'm afraid you may have jumped into the deep end (with some sharks) for your first swim.

[OwO]

Find the JTAG pins and attach the Xilinx programmer, then in Vivado you should be able to read device type and device DNA, which you can look up on Xilinx's website to check part number and date code. It does not guarantee that the entire chip is good, but at least can tell you it's not a dud. Next thing I would do is create a very basic project in Vivado and try to program it.

For desoldering and re-soldering you need a hot plate/preheater and a hot air station. Remove the nozzle from the hot air gun for the widest air beam, set wind to maximum, and continuously circle the chip with the hot air gun. Also apply flux under the chip before desoldering.
After desoldering clean the chip with a solder wick, but DON'T use a soldering iron. Set the hot air just hot enough to melt the solder blowing on the chip, add flux, and insert the wick slowly.

[lukego]

If you're an electronics newbie than I'm afraid you may have jumped into the deep end (with some sharks) for your first swim.

Sharks shmarks. Just punch 'em on the nose, right?! 

[Fred27]

I like your attitude and I'm looking forward to the video. Good luck!

[Haenk]

(I don't think a DIY desolder, reball, solder will work - unless you own a ZEVAC or the like...)
Best is to ask a company which specializes into BGA rework - e.g. we have a company located near our office which does official repair works for quite a couple of known brand mobile phones and lots of other stuff. They *do* have the skilled people and the proper equipment for such a task. Might be a good idea to locate such a company anyway near you...

[lukego]

Find the JTAG pins and attach the Xilinx programmer, then in Vivado you should be able to read device type and device DNA, which you can look up on Xilinx's website to check part number and date code. It does not guarantee that the entire chip is good, but at least can tell you it's not a dud. Next thing I would do is create a very basic project in Vivado and try to program it.

Thanks for the advice. Currently I have a Linux box with a USB MPSSE cable but no Xilinx/JTAG programmer and no Vivado license. I'll have to see if there's some open source software for programming via JTAG using the MPSSE cable and otherwise acquire a dedicated programmer. I'll hopefully be able to use a cloud-based Vivado to create a bitstream remotely, or else get a 30-day trial of Vivado to run locally, or else shell out for a license (yikes.)

For desoldering and re-soldering you need a hot plate/preheater and a hot air station. Remove the nozzle from the hot air gun for the widest air beam, set wind to maximum, and continuously circle the chip with the hot air gun. Also apply flux under the chip before desoldering.
After desoldering clean the chip with a solder wick, but DON'T use a soldering iron. Set the hot air just hot enough to melt the solder blowing on the chip, add flux, and insert the wick slowly.

Thanks for the tips! I have a preheater and a hot air station, and I have intently watched several YouTube videos of people using the technique you described on e.g. GPUs (though often wicking up solder using an iron directly, is that really so bad?)

[lukego]

(I don't think a DIY desolder, reball, solder will work - unless you own a ZEVAC or the like...)

Specifically what do you think will cause problems when reworking with hot air? Just overheat a part of the chip and damage it? How come it's a problem for the FPGA while YouTube is full of people doing it successfully with iPhones/iPads/Macbooks/etc?

Best is to ask a company which specializes into BGA rework - e.g. we have a company located near our office which does official repair works for quite a couple of known brand mobile phones and lots of other stuff. They *do* have the skilled people and the proper equipment for such a task. Might be a good idea to locate such a company anyway near you...

I ordered four of these chips. If I kill the first then I'll be more careful with the second

[Haenk]

Specifically what do you think will cause problems when reworking with hot air? Just overheat a part of the chip and damage it? How come it's a problem for the FPGA while YouTube is full of people doing it successfully with iPhones/iPads/Macbooks/etc?

a) they won't annouce a failure
b) those chips are usually way smaller, so they are easier to get the hot air below the chip
c) professional repair guys probably did this hundreds or thousands of times before - that makes a difference

I suggest to test your skills on some scrap boards, before killing off such an expensive chip
(But it's your money, so do as you like, of course...)

[lukego]

I suggest to test your skills on some scrap boards, before killing off such an expensive chip

Indeed. I have been practicing on scrap PCBs from e.g. old Macbook Air laptops. Just removing chips with the hot air rework station and putting them back on with a soldering iron.

The next practice step will be with BGA. I don't know quite what yet. Maybe just find some working but expendable device with a BGA, remove the chip, reball it and put it back, see if it works. Buy a replacement chip and repeat until that works. This should be fun and inexpensive if done with a cheap chip.

Just right now the urgent need is to test the UltraScale+ chips as well as possible during the period when I can still send them back if they are faulty. This is up-front due diligence. Then I am happy to wait and develop my skills further before I actually desolder them.

[OwO]

The free version of Vivado won't be able to synthesize for the part, but it should still be able to detect it and show device id. Xilinx JTAG cable is about $20 on aliexpress.

I don't know which videos you watched, but my experience is most videos about soldering on youtube are utter bullcrap and teach you the exactly wrong techniques. Louis Rossmann is actually one of the "better" examples (compared to everything else on youtube that is) even though that's still not how you are supposed to solder.

Using an iron with a solder wick has a high chance of ripping pads, especially if you don't yet have a good feel of the temperature settings of your iron. Too low and the solder will solidify between the wick and the board, too high and you will displace pads. Using hot air allows you to brush the wick gently over the pads.

How do you plan to reball it? Apply paste onto the chip with a stencil and reflow? A thick stencil will be needed to get enough solder on it. The other technique that I'm experimenting with is to just skip the reballing and solder it as a LGA (apply paste on the board with stencil, place chip, reflow). If you use a thin enough stencil and get just a tiny amount of solder on each pad, you can get away with pushing down on the chip during reflow which will ensure all pads make contact. I don't know how well this works with large chips though.

[lukego]

The free version of Vivado won't be able to synthesize for the part, but it should still be able to detect it and show device id.

My understanding, not yet verified first hand, is that Xilinx provide an Amazon EC2 image with the complete unrestricted Vivado for $0.36 per hour of usage here: https://aws.amazon.com/marketplace/pp/B07WNR73X5?qid=1580369452876&sr=0-5&ref_=srh_res_product_title

Xilinx JTAG cable is about $20 on aliexpress.

Just googling it seems like I should be able to use the MPSSE cable with OpenOCD to program arbitrary JTAG devices. Yes? (EDIT: Provided I have already generated a bitstream with Vivado.)

How do you plan to reball it? Apply paste onto the chip with a stencil and reflow?

Currently I like the look of this method: https://www.youtube.com/watch?v=DVTxHx0z-wo. That's wicking up solder with an iron, placing solder balls by hand, using tacky flux as an adhesive to keep them in place while reflowing them with hot air. Crazy shit but you only live once amirite.

I have practiced this on an ECP5. I'm pretty sure that I killed that chip ($5) by applying too much hot air, not seeing that the reflow had already happened. I'll do a few more dry runs until the technique feels comfortable, and then try actually resoldering a few real chips until I succeed, and then dare to desolder the UltraScale+.

A thick stencil will be needed to get enough solder on it.

Stencil may well be a better option. I have some solder paste and some thin generic stencils but I haven't tried any yet. I'm told that excellent custom stencils can be had from Aisler.net for cheap and maybe I will take that route.

I think the reason my first attempt at reballing went badly is that the solder balls don't contain flux and don't look the way I expect when they reflow, but it could also be that I just missed it. I ended up cranking the heat to 500C and nearly burning a hole in my ESD safety mat... making my newbie mistakes!

[lukego]

Just to share my journey of learning to solder and rework, here's a picture of the 0.8mm ECP5 that I cleaned up with wick and iron, then dead-bugged on one pad with a wire pencil:



I'm quite proud of that effort!

I'm less proud of this mark on my ESD mat where I heated that same chip almost certainly to death while making my first attempt at reballing:



Here is the wire connection up close. Looks to me like it worked - the insulation melted in the solder as it was supposed to - but I didn't choose the angle well for strength:



Incidentally, although I'm learning a lot from YouTube, I'm learning much more from reading the EEVblog forum archive.

[james_s]

There is no way you are going to dead bug such a large FPGA and be able to get enough decoupling for the power rails in there and enough wires hooked up to be useful. You would have to be an insane masochist to try removing it from the board, it's already professionally soldered onto. By far your easiest and most likely to succeed path is going to be using the board you have.

How much experience do you have with VHDL or Verilog? If you are starting from zero then figure on investing several years before you have anything even approaching a need for such a large part.

You say you are a "total noob" and yet you are trying to tackle the sort of project that if someone with a PhD in EE and 5-10 years of experience in hardware design succeeded it would impress me. You are drastically underestimating the difficulty of what you are trying to do.

[lukego]

You are drastically underestimating the difficulty of what you are trying to do.

You say that like it's a bad thing?

I'm excited about having a high-end part even for simple designs. Take for example the fpga4fun ethernet packet generator (https://www.fpga4fun.com/10BASE-T3.html). That usually runs at 10MHz using a normal I/O and this makes it a throw-away toy. If you ran the same program on the XCKU15P's GTY SERDES then you would be outputting over one billion packets per second at precise intervals. That's far from a toy... if you asked a vendor like IXIA for a quote on a load generation device with that bandwidth they'd probably want a milllion dollars.

So I'm excited about this hardware and all the amazing potential that it has. It's something that I could use to do stuff that I can't do at all today with off-the-shelf cards. But I'll pursue this project in parallel with other more modest ones, like programming an off-the-shelf ICE40 board and making a custom ECP5/10G board, and I'll take it at the pace that my embryonic skills allow.

For now it will serve as an excellent vehicle for learning enough JTAG to check device identity and write a bit stream. Who knows, maybe that will even lead me to discovering faults and sending the hardware back for a refund!

[james_s]

Underestimating the difficulty of a project by orders of magnitude is a bad thing unless your goal is to waste time and throw away money banging your head against the wall. Those of us who have been around here for a while have all seen a few overly ambitious beginners crash and burn and disappear.

I got a deal on a relatively high end FPGA board back when I first started developing with them, it was still much smaller than the ones you got, small enough to be supported by the free version of the software.

I eventually ended up selling it after finding I had no need for such a cavernous FPGA and the large size just meant it took much longer to compile for it than for smaller parts.

Your biggest obstacle is going to be the fact that the free software does not support that part so you are dead in the water unless you have the kilobucks to purchase a Vivado license or manage to find a cracked copy.

The board itself that it comes on looks potentially quite usable, if you desolder it from that you may as well toss it in the trash or make a keychain because it will never function again.

[thinkfat]

You say you are a "total noob" and yet you are trying to tackle the sort of project that if someone with a PhD in EE and 5-10 years of experience in hardware design succeeded it would impress me. You are drastically underestimating the difficulty of what you are trying to do.

I'd say, a PhD in EE doesn't qualify for that kind of shit 5-10 years of experience - probably.
Deadbugging an Ultrascale+ FPGA - that sounds like a recipe for disaster. There's people wetting their pants when they think of designing a board for those :-D

Can't wait to watch!

[MadTux]

Don't these things require a lot of different voltages for correct operation, like Vcore, Vio, Vmemory etc for correct operation, applied in the correct order? With Vocre something crazy as 50A at 0.8V or something when running at full load? No way this is gonna happen dead bug style with low enough ripple and without lots of big low-ESR ceramic caps directly beneath the chip.

[thm_w]

I think the reason my first attempt at reballing went badly is that the solder balls don't contain flux and don't look the way I expect when they reflow, but it could also be that I just missed it. I ended up cranking the heat to 500C and nearly burning a hole in my ESD safety mat... making my newbie mistakes!

What is the hot air station you are using? It should never be necessary to set the temperature that high. I think max out should be around 400C or so.
This could mean you either do not have enough preheat or the wattage of station is not high enough.

The best 863 clone stations are good value, and not that expensive. ~$180 for 1200W. Or the genuine one is ~$300.
For a FPGA of this size, I would speculate you need >700W but thats just a guess.

[YetAnotherTechie]

If you want to learn the chips, use the boards as they are, they (are supposed to) work, and there would be no variables beyond your coding skills. Make your goal achievable by setting it to: being able to communicate using the existing 2(?) lines of GBe serdes that drive your interest and that are already there completely wired for you. Once you have that, and if your interest is still there, *research* how to connect the fpga to the real world, there is no point removing them from a working system if you don't have an adequate pcb to solder them to (i'm not talking about the soldering part). For someone skilled in high-speed digital AND low voltage/high current/low ripple power suply, the pcb design for that beast is on itself a work for several months. Skip the frustration of not being able to do the ethernet stuff because you fail at the pcb design stuff.

[DaJMasta]

As mentioned repeatedly, do not even attempt to dead bug prototype this chip.

Even if you can work like a machine night and day to absolutely perfect results the very first time (and you cannot, make no mistake), all of your decoupling caps will be too far away to have any effect, all of your controlled impedance traces and differential pairs will be mismatches, and you will never see anything close to the full performance of the device if you can get it to boot at all.  And this is if you can actually execute 1500 pins worth of fine pitch dead bug soldering without accidentally mashing them and ripping balls off the chip, coughing and getting dirt in a place that cooks into a short, or otherwise.


With packages like this reflow is the only option.  Vapor phase soldering may not be a requirement, but at the very least you need a board with a footprint, a preheater, a good hot air station, a BGA stencil appropriate for it, good solder paste, and good flux, ideally stuff designed for this kind of soldering.  That's a bare minimum even with good soldering skills.

I'm not the kind to tell people not to do something because they can't, but it is frankly foolish to attempt such a huge project without understanding the constraints.  1500 fine pitch contact points (a few hundred of which are probably mandatory just to power the device up) should dissuade most from attempting it... but the physical limitations of such a layout are too great to overcome with just that kind of persistence, unless your target clockrate is just a fraction of what it can operate at.  The reason decoupling caps are put on the bottom of the PCB for big BGAs isn't just to save space, it's because the loop area at high frequencies to and from the cap can make it functionally nonexistent if it was put on the top of the board, next to the chip, in many cases.



Anyways, see what you can do with the chips on the boards they are already on - it will be much easier with decoupling, power, and soldering taken care of, and then you can try to find/breakout IOs and things to reprogram at least somewhat to your desires.  When you get some comfort with dealing with the chip and you have at least one that works, maybe try fabbing a carrier board for the chip in a CAD package.  It will be an immense amount of work, dozens of hours, but with a board from a PCB fab with tight enough tolerances to accommodate such a big chip, you may actually be able to keep decoupling caps and rail generation close enough to use the chip normally and take advantage of all of its IOs.  In the mean time, trying similar work with MUCH smaller chips would probably help develop the skills and understanding needed to deal with such a beast.

To that point - maybe an early project would be to try and dead-bug some DDR memory.  I'd be impressed if you could even get it to work (and that's not a slight on your skills, this comes down to layout restrictions and precision).  Again with trace length matching and impedance control, even a chip with a tiny fraction of the number of balls and operating at a lower frequency, I don't think that memory would be usable at rated speed by a normal ARM MCU in a prototype situation without a PCB.  There's a possibility, sure, and it may be a fun exercise to try..... but for the big timesink it will be, at least it won't cost hundreds of dollars and it should give you an idea of how difficult the task you're describing is.