Guide: Getting Xilinx ISE to work with Windows 8 / Windows 10 (64-bit)

[robca]

PSA: There seem to be two new issues in newer versions of W10.

During installation, it hangs around 92% when checking Webtalk. What worked for me, was to terminate webtalk32.dll (under the ISE setup app in Task Manager), wait a moment, then terminate webtalk.dll. Setup ends seemingly successfully at that point

And ISE is broken again in Windows 10, but luckily someone smarter than me figured out is was due to an interaction with the "fake" WSL network shares causing problems to the network shares enumeration. The fix is either to remove WSL (not an option for me) or patch a file

https://www.exxosforum.co.uk/forum/viewtopic.php?p=95891&sid=7526ed0d6748614255e95d07bf38ce0b#p95891

User czietz patched libPortabilityNOSH.dll to always fail network shares enumeration and provided a fixed file (renamed to libPortability.dll, to solve the Smartheap problem already mentioned). I copied the patched files into both locations C:\Xilinx\14.7\ISE_DS\ISE\lib\nt64 and C:\Xilinx\14.7\ISE_DS\common\lib\nt64 and everything seems to work fine

Downloading a binary from a random forum is usually risky, but the patch is limited to a few bytes

0001AB30: BB 8B
0001AB31: 60 F8
0001AB32: 09 BB
0001AB33: 00 60
0001AB34: 00 09
0001AB35: 48 00
0001AB36: 8B 00
0001AB37: CB 48
0001AB39: F9 CB

0:  8b f8                   mov    edi,eax
2:  bb 60 09 00 00          mov    ebx,0x960
7:  48 cb                   rex.W retf

And if I interpreted the binary correctly, it simply returns early from a call, so pretty safe

Copying the file here, just in case. Hope it's ok with the original author

[suudy]

Sorry to resurrect this old thread.  We are working through the forced Windows 11 installs we are getting on new PC's at work now, and I've found this while digging on how to get ISE to work.

Downloading a binary from a random forum is usually risky, but the patch is limited to a few bytes

This does appear to work.  But I'm not sure about the explanation below.

0001AB30: BB 8B
0001AB31: 60 F8
0001AB32: 09 BB
0001AB33: 00 60
0001AB34: 00 09
0001AB35: 48 00
0001AB36: 8B 00
0001AB37: CB 48
0001AB39: F9 CB

0:  8b f8                   mov    edi,eax
2:  bb 60 09 00 00          mov    ebx,0x960
7:  48 cb                   rex.W retf

And if I interpreted the binary correctly, it simply returns early from a call, so pretty safe

The above disassembly doesn't seem quite right.

First, here's the disassembly, based on defuse.ca's online disassembler, of the original libPortabilityNOSH.dll at offset 0x1ab30:

0:  8b f8                   mov    edi,eax
2:  bb 60 09 00 00          mov    ebx,0x960
7:  48 8b cb                mov    rcx,rbx
a:  e8 45 04 05 00          call   0x50454

And here's the disassembly based on the attached DLL file:

0:  bb 60 09 00 00          mov    ebx,0x960
5:  48 8b cb                mov    rcx,rbx
8:  8b f9                   mov    edi,ecx
a:  e8 45 04 05 00          call   0x50454
f:  48 8b f0                mov    rsi,rax

Note that the call is a rel32 (32-bit offset).  But the point here is that it doesn't return from the function early.  It only changes the registers before calling whatever is at 0x50454.

The only difference here is that edi gets loaded with edx, rather than eax.  Assuming the Microsoft x64 calling convention (this is a 64-bit DLL), the arguments are pushed RCX, RDX, R8, R9.  RAX is non-volatile and is whatever is in the caller's state.

I'm not sure what this does, or how it does it.  But it doesn't appear to return early.

[suudy]

Ok, I've done some work and have a new version of the DLL that I think makes sense.  And importantly, *it works!*

I ran the existing libportabilityNOSH.dll through Ghidra and decompiled things.  The previous patch appears to bypass the code after a call to WNetOpenEnumA by manipulating the return value.  This isn't clean, and seems to mess with arguments to the function immediately afterward (call to "new").

So, where the original patch did it's work was at file offset 0x1ab30.  Based on the DLL headers, this maps to a load address offset of 0x1000, and the .text section starts at file offset 0x400.  This means the address of the patched code is at a virtual offset of 0x1b730.  A disassembly/decompiling of the code yields:

18001b729 8b ca           MOV        ECX,EDX
18001b72b e8 2a 04      CALL       MPR.DLL::WNetOpenEnumA                           DWORD WNetOpenEnumA(DWORD dwScop
                 05 00
18001b730 8b f8           MOV       EDI,EAX
18001b732 bb 60 09      MOV        EBX,0x960
                 00 00
18001b737 48 8b cb      MOV        RCX,RBX
18001b73a e8 45 04      CALL       MSVCR90.DLL::operator_new                        void * operator_new(__uint64 par
                 05 00

Note that the patch modified the code *after* the call to WNetOpenEnumA.  But it did so in an odd way.  The return value from WNetOpenEnumA is in EAX.  The original code saves this in EDI, then sets up a call to "new".  Here's what original patch does:

18001b729 8b ca           MOV        ECX,EDX
18001b72b e8 2a 04      CALL       MPR.DLL::WNetOpenEnumA                           DWORD WNetOpenEnumA(DWORD dwScop
                 05 00
18001b730 bb 60 09      MOV        EBX,0x960
                 00 00
18001b735 48 8b cb      MOV        RCX,RBX
18001b738 8b f9           MOV        EDI,ECX
18001b73a e8 45 04      CALL       MSVCR90.DLL::operator_new                        void * operator_new(__uint64 par
                 05 00

This effectively only tweaks the return value of WNetOpenEnumA.  That is, it fails to store the return value of WNetOpenEnumA into the local stack frame, leading to the later checks to fail, bypassing much of the code.  This works, but does not completely bypass the function call.

Instead, I have proposed the following 4 byte patch which does what was suggested in the earlier email.  This patch returns earlier from the function, and fixes the checksum of the DLL.  Here's the change:

18001b700 c3              RET
18001b701 90              NOP

That's it!  And it works.

So, there are only 4 bytes in the entire file to change:

00000151 D1 54
00000152 0D 4B
0001AB01 40 C3
0001AB02 53 90

Attached is a bsdiff patch file.  See this link for details on how to apply it to libPortabilityNOSH.dll.

 libPortabilityNOSH-patch.zip (0.34 kB - downloaded 41 times.)