Network test equipment - Broken FPGA??

[johnmx]

Hi,

At my work (public research group) we have an expensive network test equipment bought in 2009 that stopped working after being unused for 3 months. The manufacturer (IXIAcom) doesn’t give support anymore and the only way is to buy a new mainframe. But I want to try to repair it before giving up.
I have no experience with FPGA’s, so I’m asking your personal opinion about where could be the fault.

The system
The main frame is made of a complete PC, a control board and power supplies. I was able to make a block diagram (attached picture) just by looking to the system.
The PC seems fine, including its RAM memory. I double checked with a special memory analyse tool.

The Error
When the system boots, it gives a timeout error while downloading the FPGA’s firmware. I can open the firmware file (b0019B.hex) on the host PC with notepad and it seems fine.

The booting process
According to the created log file (original text inside double quotes) and my personal analysis, the system performs the following (please see the attached picture):
- Power-up
- Establish connection with the MFPGA (XC2S150) via TCP sockets
- Get MFPGA version: "MFPGA version 31". This FPGA was programmed automatically by the CPLD during boot. This firmware is located in the PROM XC18V01.
- Program the MFPGA with a newer firmware. The SRAM should be used as a buffer:
    - "Downloading MFPGA via SRAM Interface"
    - "Verifying MFPGA download via SRAM Interface"
    - "Configuring MFPGA via SRAM Interface"
- Get MFPGA version: “MFPGA version 52”
- Enable chassis power   
- Get power supplies status and test all 8 huge fans by checking their maximum rpm.
- Program the big FPGA's (XC2VP7):
    - “Download C:\Program Files\Ixia\(...)\Fpga\b0019B.hex”
- But after some seconds it gives an error:
    - “FAIL: Memory read response timeout error at offset 00151F40 with group 0 and channel 0, Hdr: 0001 2002 0001 00000000”
- After some seconds this error is reported again 7 times. After that the system cancels the boot process and shutdown.

What I’ve done so far
I replaced the SRAM chip (71V124) but the error persists.

The big FPGAs (XC2VP7) are expensive and difficult to replace. But I can ask a local company to do it.

In your opinion, do you think the problem is a broken FPGA?

Thanks.

[XOIIO]

I know little about these things, but does the FPGA perhaps pull something from a nearby eeprom or prom? Maybe try to read/dump the contents of that to check that they are intact?

[mikeselectricstuff]

Have you checked all the supply rails?

[all_repair]

Get a new supply if it is not too expensive and test.  More chances that the supply cannot source the current needed during the power cycling than the digital circuitries.

[miguelvp]

If it's not the rails.

Any JTAG connectors to the fpga?

Also what is the speed grade, can't see it from the thermal grease on the lower left side of the chip, If it is 5C or 6C then they are not too expensive.

XC2VP7-5FF896C
XC2VP7-6FF896C

Didn't look in octopart but findchips shows Quest having them for $114 & $225 respectively.

http://www.findchips.com/search/XC2VP7


Datasheet:

http://datasheet.octopart.com/XC2VP7-6FF896C-Xilinx-datasheet-13696476.pdf

Boundary Scan configuration and other configurations are discussed on page 56

If you need a cheap and compatible programming adapter:

http://www.wayengineer.com/wx-platform-cable-usb-p-4782.html

[xygor]

Does the "done" pin from the fpga go hi at any time?  This signal should make its way to the CPLD if it's more convenient to probe there.  If there is a programming header for the FPGA and it is NOT jtag, then "done" will make an appearance there also.
Is the CPLD a Xilinx SystemACE or something else?

[johnmx]

I found that one of the biggest FPGA’s is another model but from the same family, a XC2VP20.

The PFPGA heats a little bit when the error happens, so I guess that this is the one that is being programmed and is creating the error. The SFPGA is always cool.  I think that P and S stands for Primary and Secondary. I removed the heat sinks and used a Flir Thermal Imaging camera.

I know little about these things, but does the FPGA perhaps pull something from a nearby eeprom or prom? Maybe try to read/dump the contents of that to check that they are intact?

The only static memory on board is the PROM with the main FPGA’s firmware (MFPGA) not for the PFPGA neither the SFPGA.

Have you checked all the supply rails?

There are three LT1963 (LDO, 1.5A) for each big FPGA. All of them are fine with 2.5V, 2.5V and 1.7V. I guess that 2.5V is for the Vaux and 1.7V for Vcco. There is also 1.5V on several big tantalum capacitors, just don’t know where the regulator is… So, the supply rails seams fine.

Any JTAG connectors to the fpga?

Also what is the speed grade, can't see it from the thermal grease on the lower left side of the chip, If it is 5C or 6C then they are not too expensive.

There are three connectors, each very close to each FPGA (see attached picture). Doesn’t look a typical JTAG connector if such thing exist...
Bellow the FPGA I found the via of the TMS signal. But it doesn’t appear on these connectors.

The speed grade is 6C for both. The model numbers are XC2VP20-6FF896C and XC2VP7-6FF896C and they are available online.

[johnmx]

Does the "done" pin from the fpga go hi at any time?  This signal should make its way to the CPLD if it's more convenient to probe there.  If there is a programming header for the FPGA and it is NOT jtag, then "done" will make an appearance there also.
Is the CPLD a Xilinx SystemACE or something else?

I was able to check the DONE signal through the underneath via. This signal is at low level when the system is turned on and goes high after 1s or so. Then it goes low again when the download firmware message appears. I think it is the expected behavior.
Since the error cancels the boot process, this signal never return to high.
So, I think the problem is related to the FPGA programming.

The CPLD is used to program another FPGA.

[nctnico]

DONE should go high after the FPGA has been programmed. I'd look at the serial programming inputs of the FPGA to see if they receive a bitstream; take a close look at proper edges and signal levels. Maybe it is just the data which doesn't arrive at the FPGA.

[johnmx]

I replaced both FPGAs XC2VP20 and XC2VP7 and the equipment is working again.  I bought them from Quest Components.
It’s impossible to know why they broke…

Happy ending.