Finding fault in early 90's ISA PC Card

[rx8pilot]

Is there some tantalums I see. Are they ok?

They are not shorted and not burned. If they are weak, I don't think they would cause a full card failure - but who knows. They appear to be just global filters not critical to any timing or bypassing.

I can try looking into the software with disassembler to check the cause of that error message if you wouldn't mind to share the sw.

The software is hardware keyed which will make that offer impossible. I was wonder though, if there is a utility of some kind that can monitor to the ISA bus so see what data is passing through. I cannot tell yet if it is doing anything or nothing. Since the computer behavior is the same regardless of whether the card is installed or not is weird to me. The drivers appear to be made with WinRT Toolkit which is a shortcut to make a driver if you don't know how to make drivers. The relative simplicity of this card would make it a good candidate. Anyway, the OS reports everything as OK and working properly even when the card is absent. There are no errors, flags, additional warnings that the card is missing, etc.

No IRQ conflicts that are reported by the OS. I will try to replace more of the chips on the ISA bus side of the card to see if I get lucky.

[abyrvalg]

I don't need to run that sw to analyze it (doing static analysis in IDA Pro instead). No data files required also, just the exe and driver sys.

You've told that entire setup was converted from DOS to XP. Was the main sw replaced with some never Windows version or it is the same old DOS app running under XP now?
For "DOS under XP" case the driver should be pretty dumb: just make card's IO ports range accessible to DOS app which does all IO. No easy ways to sniff IO in this case.
But there are some ways for native XP app: first you can try DebugView https://technet.microsoft.com/en-us/library/bb896647.aspx to see if the driver outputs something (some drivers do). Next is a heavier tool that shows all activity between apps and drivers - http://www.osronline.com/article.cfm?article=199

[rx8pilot]

I don't need to run that sw to analyze it (doing static analysis in IDA Pro instead). No data files required also, just the exe and driver sys.

You've told that entire setup was converted from DOS to XP. Was the main sw replaced with some never Windows version or it is the same old DOS app running under XP now?
For "DOS under XP" case the driver should be pretty dumb: just make card's IO ports range accessible to DOS app which does all IO. No easy ways to sniff IO in this case.
But there are some ways for native XP app: first you can try DebugView https://technet.microsoft.com/en-us/library/bb896647.aspx to see if the driver outputs something (some drivers do). Next is a heavier tool that shows all activity between apps and drivers - http://www.osronline.com/article.cfm?article=199

I see what you are saying now. The original DOS software is no longer used at all. The software is a completely new application written for XP and I believe they used WinRT Toolkit to make the driver for the old ISA card. There are a number of other I/O's that have been consolidated into a single USB that is broken out at a custom interface card that has 4 FTDI serial interfaces (which all work well). There is also one dedicated RS-232 being used as well.

The entire system was pre-built including all software and configurations by the company that wrote the software/drivers. At least I know that the entire hardware system has, at one time, worked well together. I got the docs on the SBC (single board computer) which is a Portwell ROBO-8712EVG2A with a Pentium 4 CPU and 1GB RAM. It is attached to a Portwell PBP-14A7 backplane with PCI and ISA slots. The SBC has a built in PCI-ISA bridge, but I am not sure if that bridge needs a specific driver,service, etc to be active. I do not see any obvious devices or services that seem like they have anything to do with an ISA bridge. Any idea if that is a hardware element that runs on its own or is software needed to get it connected to the IO of the PC?

I now have another card (function unverified) that behaves the same way. In fact, the PC responds the same way whether the card is present of not. The System Information says everything is OK even if there is no card attached. Device Manager does not show any errors when the card is absent either.

I am going to look at the links you provided, hopefully they will provide some clues about the status the ISA bus in general.

[rx8pilot]

DebugView output:

Line 40 shows the "HS3L Task"
HS3L refers to the ISA Card (Model MVS922). Not sure what the significance of the mapped memory address 0x258 (guessing that is an address)

[abyrvalg]

The PCI-ISA bridge should be brought up by BIOS and remain active, nothing special there.

For non-PnP card it is normal to show itself in device manager when it is removed actually. There is no generic way for OS to check the presense of such card or detect it's IO/IRQ params (they are specified by driver installer instead). It is driver's responsibility to check for card presense (in some card-specific way, read some "magic" value from some register usually), but it is optional (or the hardware is too simple for that).

That repeated "re-sending ABORT" can be the cause - something like no reply to some ABORT message from the remote side of RS485, but it is hard to say where is the problem - transmitter/receiver/cable/remote side disconnected/powered down/broken/etc. Observing RS485 lines with a scope/LA/or even a multimeter (is there any voltage change during this activity?) can show more.

I wonder why they didn't replaced this board with some USB-Serial converter (just another FTDI chip + ADM485)?

[rx8pilot]

I wonder why they didn't replaced this board with some USB-Serial converter (just another FTDI chip + ADM485)?

I think this is because they don't know what it is doing. It is paired with an industrial machine vision processor from the early 90's and it seems to be doing slightly more than just a simple RS-485 driver. It is buffering and/or manipulating the data in some way (or so it seems).

More troubleshooting notes that could be relevant. There was originally a PCI USB card in the chassis, but the system will not boot with it in there. Take it out, all is well, put it in - no boot. Also some strange (onboard) NIC behavior. The NIC's seem to load ok but when I plug in an ethernet cable and it tries to get an IP - the whole system freezes. Not a blue screen crash - it just freezes. No caps lock, mouse movement, etc requiring a reboot. If I don't plug in the Ethernet, it works fine. It could be related to a bus problem or not. On board USB works, COM1/2 work ok, and PS/2 mouse and KB work ok.

[Towger]

Might be time to track down another 'computer'.  Is the NIC card PRO or PCI?
Would you get away with a standard P4 board with PCI and ISA slots?

Edit.. Another thought.
Is it possible the card is working, but it not getting a response in turn, from whatever it is connected to.

[abyrvalg]

The NIC behavior can be a sign of resources conflict. Try two tests:

1. Remove as many removable cards as possible (including MV922 of course) then power up and connect the network cable - will it freeze the system as usual?

2. With all cards back disable the NIC in BIOS (is there such setting?), then run the pick&place app - no change?

[rx8pilot]

Might be time to track down another 'computer'.  Is the NIC card PRO or PCI?
Would you get away with a standard P4 board with PCI and ISA slots?

Not sure, the hardware configuration is pretty specific to the software. Hard to say....

Is it possible the card is working, but it not getting a response in turn, from whatever it is connected to.

It is possible. Based on the error messages I am truly guessing. I could hook up a scope to the RS485 RX and TX lines and see if any activity is there. If there is some Tx and no Rx, could be a clue.

The NIC behavior can be a sign of resources conflict. Try two tests:

1. Remove as many removable cards as possible (including MV922 of course) then power up and connect the network cable - will it freeze the system as usual?

2. With all cards back disable the NIC in BIOS (is there such setting?), then run the pick&place app - no change?

The MVS922 is the only card in the machine. All other functions are integrated in the single board PC. Graphics, Dual LAN, IDE controller, USB, Serial, Parallel........
I will dig through the BIOS to see if the NIC can be disabled. For the moment, I disabled them in the device manager but that does not seem to change anything yet.

[T3sl4co1l]

No idea what the later bus standards (PCI, etc.) do about it, but hard freezing is a traditional 'panic' mode in the IBM-PC when /NMI is pulled on the ISA bus.  This is usually reserved for "shit's broke" signals, and the usual BIOS ISR (interrupt service routine) is an infinite loop (JMP +00 or something like that).  So interrupts remain disabled, the processor doesn't read any memory but the BIOS that it's looping on, and peripherals are left alone.  Since the keyboard ISR isn't used, CTRL+ALT+DEL will not clear this fault, and a hard reset is required.

I would be surprised if this sort of signal and code path is still present, but it could be that something vaguely similar is happening.  The traditional "panic" of a 386+ user-kernel system (typical of *nix and Windows NT+) is an escalating processor fault (e.g., page fault, general protection fault (GPF)) that causes the kernel code itself to crash, triggering a reset (and on Windows, a BSoD, at least briefly before the reset vector is executed).

Could very well be that the driver is so poorly written that it doesn't play nicely with any of the systems integrated on the SBC.  I have no idea what driver or kernel programming is like in Windows, so... could very well be, they don't know either..

Tim

[poot36]

Try booting the computer off a linux or windows live cd to see if the network card works then.  If it does then there is a software issue that needs to be looked at.  Try running chkdsk /f /r and sfc /scannow (you will need a windows install cd for the sfc check) and see if that fixes the problem.  If that does not work you can try running tweaking.com's system repair tool but make sure to make a registry and system restore point (and maybe a image of the hard drive) before you run the repairs making sure that you click the select all button for the repairs.  If it still locks up the boot disk then there is either a bios configuration error or corruption or a hardware issue.  Record all of the bios settings and remove the bios battery and power to the computer and short out the battery holders contacts for 30 seconds then replace the battery and repower the system and enter in the bios settings again and see what happens.  Also check the voltage of the bios battery as well.  You can also try a bios update as some bioses hold the configuration data for the network card in them as well.  I have personal experience in a very low bios battery causing a industrial pc to not even power on and have also seen linux boot disks fix totally dead usb ports on a Dell xps system (the bios would not even see the usb keyboard, I measured the d+ and d- of all the ports on the system and they were hovering around 3.3V or so so they were not been initialized properly by the software in the bios but the linux boot disk was able to get it out of that weird situation).

[abyrvalg]

Old IO port manuals says that NE2000-style NICs can be configured to 240-25F port range - that overlaps 258 where your HS3L card lives (according to DebugView). I would recommend looking at NIC's settings carefully (is there any option in BIOS? check IO range in windows device manager also)

[Shock]

Disable any unused BIOS functions that may chew space on execution. e.g. PXE booting. That may cause failure to boot and a hang situations if some other BIOS function is compromised. Especially if there has been a defaults restore, backup battery failure, or board changed.

[Rasz]

try another P3 era computer

[rx8pilot]

I think this repair attempt has met the point of diminishing return. I replaced the majority of the logic, the RAM, and the RS485 drivers with no forward progress. The parts are cheap, but my time limit is probably up. Next stage is get one from the company that supports the machine at a high cost, but less than a useless pick and place machine. They will support the project only after I buy the card from them and they seem to have the secrets since they wrote the software and built the PC setup.

Arrggg......I was hoping to get lucky. Thank you all for the help.

[senso]

They have you hanged by the balls..
There is no support/warranty from the company that sold you the machine, or is it the same company that is now "selling" you the card for 500$?

Given the price and rarity of the machine and what makes it tick I would be very worried that you might have similar problems in a couple months and they will charge you thousands to apply duct-tape..

[Rasz]

I think this repair attempt has met the point of diminishing return. I replaced the majority of the logic, the RAM, and the RS485 drivers with no forward progress. The parts are cheap, but my time limit is probably up. Next stage is get one from the company that supports the machine at a high cost, but less than a useless pick and place machine. They will support the project only after I buy the card from them and they seem to have the secrets since they wrote the software and built the PC setup.

Arrggg......I was hoping to get lucky. Thank you all for the help.

look for a hackerspace in your vicinity, offer straight up cash/time on fixed machine HS donation for help, let people that do this sort of thing _for fun_ do their magic.

[coppice]

Have you checked the condition of the edge connectors? Its very common for the connectors on 90s era boards to be so brittle by now that they crack with barely a touch. The cracks aren't always obvious unless you look carefully, but can remove almost all pressure from the contacts.

[amyk]

ISA is slow enough to probe with a scope or LA... and since the card is mapped to a single port, capturing the communication to/from it should be relatively easy.

How many layers is the card? It shouldn't be that hard to RE and derive a schematic. The logic arrays (likely GAL/PAL) might be a bit more troublesome but it shouldn't be hard to figure out their possible function once the components connected to them are determined.

I have a feeling that the problem is NOT in the card itself.

It's not uncommon for companies like this to lead you on a wild goose chase...

[rx8pilot]

Have you checked the condition of the edge connectors? Its very common for the connectors on 90s era boards to be so brittle by now that they crack with barely a touch. The cracks aren't always obvious unless you look carefully, but can remove almost all pressure from the contacts.

I looked at them under a microscope early in the investigation and did not see anything unusual. The traces coming from the gold fingers are senselessly small though. A tiny crack could be beyond what my low power scope can see. I should check again with a higher power.

ISA is slow enough to probe with a scope or LA... and since the card is mapped to a single port, capturing the communication to/from it should be relatively easy.

How many layers is the card? It shouldn't be that hard to RE and derive a schematic. The logic arrays (likely GAL/PAL) might be a bit more troublesome but it shouldn't be hard to figure out their possible function once the components connected to them are determined.

I have a feeling that the problem is NOT in the card itself.

It's not uncommon for companies like this to lead you on a wild goose chase...

It may or may not be the card. The investigation of the card and the PC have not revealed a smoking gun yet. The software and drivers are as closed as possible and no information is publicly available. As much as it sucks, this is how companies stay in business - they have secret knowledge that someone else needs to be able to keep their business going. I would not fault PPM for operating this way, but I would prefer to stay away from closed systems as much as possible. In this case, I got a fixer upper Pick and Place machine for $5k. If I have to put a few $k into fixing it, so be it. The reality is that it will make it value make each and every month - and PPM knows that. I could have chosen a $5k low-performance TM-240a or spent north of $50k on various low-end NEW vision based systems out there. The Quad, if working, will be great and in all likelihood will be fully functioning for under $10k.

Not too discouraged......yet. ;-)

[krivx]

Have you checked the condition of the edge connectors? Its very common for the connectors on 90s era boards to be so brittle by now that they crack with barely a touch. The cracks aren't always obvious unless you look carefully, but can remove almost all pressure from the contacts.

I looked at them under a microscope early in the investigation and did not see anything unusual. The traces coming from the gold fingers are senselessly small though. A tiny crack could be beyond what my low power scope can see. I should check again with a higher power.

I wouldn't bother. Plug the card into the mainboard and check for continuity between vias/pins on both the card and the mainboard. Should take 10-20 minutes and will be much more reliable than a visual inspection.

[rx8pilot]

SOLVED!

It was indeed a connection issue, just not with the ISA card at all, not an OS issue, not a BIOS issue. I re-seated the SBC on its back plane along with the RAM - FIXED. I am so embarrassed I did not try that earlier. Nothing was ever really broken - it was just not connected.

I went down many paths looking for the root cause in the most complicated places. Software, bugs, replacing chips, etc. In the end, the solution was about 30 seconds and free. 

[PA0PBZ]

SOLVED!

 

[krivx]

SOLVED!

I went down many paths looking for the root cause in the most complicated places. Software, bugs, replacing chips, etc. In the end, the solution was about 30 seconds and free. 

Hey, you could have forked out money for replacement parts that wouldn't have helped, this sounds like a victory to me.

[Towger]

Good to hear. Might be worth while cleaning all the motherboard connectors etc. You now have two of these cards, are there many others required to build a second backup computer, using a 'standard' ISA/PCI motherboard.
BT any chance of more photos?