EEVblog Electronics Community Forum

Products => Computers => Topic started by: Eelt on January 06, 2022, 12:06:03 am

Title: Powering on Server with proprietary but documented power connector
Post by: Eelt on January 06, 2022, 12:06:03 am
Hi, I'm new to the forums, but I have been watching EEVblog for a few years now and thought that this would be a good place to ask my question/get a second opinion about powering on a server that uses a proprietary power connector since this community seems to be more oriented/knowledgeable with electricity :).

I bought one of these: (https://www.ebay.ca/itm/333986600758 (https://www.ebay.ca/itm/333986600758)) first-gen 96-core ThunderX servers made by Gigabyte with the intention of running some sort of Hypervisor (if I can get everything up and running) or Linux Distro (for server usage). This is a single node that would've been part of a 4-node system, however, it's not a true "blade" server as each node still has its own IO (4x USB 3.0 5 Gbps, 2x QSFP+, 2x VGA, 1x Serial, 1x IPMI per node) only connecting to what seems to be a shared power backplane/PSU (originally the system would have 2x 1600 W PSUs which Gigabyte claims is fully redundant, aka 400W/node). 

Here's Gigabyte's official page for the full system, if you click on the specification tab you can see the full details of the full system: https://www.gigabyte.com/Enterprise/ARM-Server/H270-T70-rev-110 (https://www.gigabyte.com/Enterprise/ARM-Server/H270-T70-rev-110)

Fortunately, I'm not the first person who has bought one of these nodes (and if I was I wouldn't've), others have figured out what the fingers on the nodes Power Daughterboard do, and looks like at least 2 people have managed to get this system working.
Here's a labelled image of what the power board looks like and what the fingers do, done by fifteenhex on GitHub:
 
Top: https://github.com/fifteenhex/crappy96corearmserverexperiments/blob/master/power_distribution_board/front.jpg (https://github.com/fifteenhex/crappy96corearmserverexperiments/blob/master/power_distribution_board/front.jpg) 
Top (Direct): https://raw.githubusercontent.com/fifteenhex/crappy96corearmserverexperiments/master/power_distribution_board/front.jpg (https://raw.githubusercontent.com/fifteenhex/crappy96corearmserverexperiments/master/power_distribution_board/front.jpg)

Bottom: https://github.com/fifteenhex/crappy96corearmserverexperiments/blob/master/power_distribution_board/back.jpg (https://github.com/fifteenhex/crappy96corearmserverexperiments/blob/master/power_distribution_board/back.jpg)
Bottom (Direct): https://raw.githubusercontent.com/fifteenhex/crappy96corearmserverexperiments/master/power_distribution_board/back.jpg (https://raw.githubusercontent.com/fifteenhex/crappy96corearmserverexperiments/master/power_distribution_board/back.jpg)

So my *real* question is, how do I actually give these fingers power? I saw an Anandtech review with 2x CN8890s (the CPUs that are in this server) at full load can use up to 450 W, so I'd assume say taking my Dupont Wires for my Arduino kit would be a really bad idea (not sure what kind of wiring I could buy off of Amazon for this). I've been thinking about using my spare EVGA 750 G5, but I'm concerned about theoretically loading a single 12V pin with 37.5 A of current. Also, plugging in multiple 12V leads to the same finger, since it would be parallel, wouldn't it result in the server receiving >12V on the finger? Also, the 12V Standby (small finger right next to the big ground finger) is a problem since ATX PSU's only have a 5V standby (probably why the image has 5V STB crossed out and replaced with 12), so I'd probably use another 12V pin from either the 24-pin or a PCIe for the standby finger and just have the PSU on 24/7 (aka never shut the server off once it's ready for long-term use)? For the PSON, would that just be connecting the second last small finger (looking at the bottom image) to the PSON pin on the ATX 24-pin (Pin #16)? As for the other small fingers, I'm not really sure what they're supposed to do or if they're necessary to start the system.

I came here to see if what I'm thinking is sane/looking at it the right way, or if anyone has a better solution to this problem?

TL;DR: What wiring should I buy for this, if the primary 12V finger could draw a maximum of 450W (37.5A@12V)? What should I use as my power supply if I shouldn't jerry-rig a spare ATX one (and if I should jerry-rig it, any good recommended/preferred pins I should use for 12V, ground, etc)? What's the best way for me to handle the 12V Standby for the IPMI/BMC, just have the PSU on 24/7?
Title: Re: Powering on Server with proprietary but documented power connector
Post by: ve7xen on January 06, 2022, 12:43:34 am
Quote
So my *real* question is, how do I actually give these fingers power?

Cheap and nasty option is just to solder some decently heavy gauge wire to them.

They are designed to mate with card edge connectors. Looks like the board has markings A1-A14 and B1-B14, so I guess it is a 28-pin connector, probably 4mm pitch (you would be wise to measure it...). Usually rated for 3A per contact, and it looks like the 12V pad is made of either 4 or 5 contacts on both sides, so total capacity of 24A or 30A. If these assumptions are correct, the mating connector would be Hirose CR7C-28DA-4DS, I think. Mouser: https://www.mouser.com/ProductDetail/Hirose-Connector/CR7C-28DA-4DS?qs=sGAEpiMZZMvShe%252BZiYheiqTffYEOnDp8kWyRt6obIGA%3D (https://www.mouser.com/ProductDetail/Hirose-Connector/CR7C-28DA-4DS?qs=sGAEpiMZZMvShe%252BZiYheiqTffYEOnDp8kWyRt6obIGA%3D)

I would use the solid copper core of 12 or 10 AWG house wiring available at the hardware store to parallel the pins at the connector, then solder all the power wires from one of the power supply's VGA cables to that. How much current can safely be carried is a function of acceptable temperature rise. You can find 'ampacity' charts online. The PSU wiring is probably 18AWG and there are 4 conductors for +12 and GND on one of those connectors. Should be good for 40A.

Do you know the consumption of the BMC on the Vsb line? If it's fairly low, use a boost converter on the +5Vsb to boost it to 12V. Otherwise I would probably just use a separate 12V wall-wart for that. It will probably be more efficient at low load than the full ATX supply.

There's a good chance PSON can connect to the ATX signal for the same purpose, enabling powering on the server with the front panel button. But you'll need a bit more information about the board's PSON scheme to confirm (or just try it). You can probably also ignore this pin entirely and attach the ATX PSON to a separate switch and power it on that way.

Edit: Looking more closely at the photos, it looks like you might be able to connect the ATX supply directly to those ATX-like connectors...
Title: Re: Powering on Server with proprietary but documented power connector
Post by: Eelt on January 06, 2022, 02:29:51 am
Thanks for the reply!

I'm (obviously) not an electrical expert, but this sounds kind of what I was expecting.

Quote
Looking more closely at the photos, it looks like you might be able to connect the ATX supply directly to those ATX-like connectors...
They are 18-pin connectors that attach to the mainboard. The mainboard has 2 of these, one is left unpopulated. The node I have the left model so the powerboard is on the right and uses the mainboard's right side 18-pin. I assume the one on the left is for the right model so that Gigabyte only has to make 1 mainboard for both node types. HP has an 18-pin connector as well, but there's no guarantee that it's the same pinout. There's no EPS-like connector, so the CPUs must be pulling power from the 18-pin (along with memory).

There's also a 5-pin "PCIe-like" connector on the Powerboard that goes straight to the SATA backplane on the front... it would be nice if I would be able to use it. Additionally, the fans also run off the daughterboard, so overall I thought it'd be easier just to figure out how to get power to the fingers rather than dealing with SATA power, Fans, and whatever other unforeseen functions may not be running off of the main 18-pin.

Quote
Do you know the consumption of the BMC on the Vsb line? If it's fairly low, use a boost converter on the +5Vsb to boost it to 12V. Otherwise I would probably just use a separate 12V wall-wart for that. It will probably be more efficient at low load than the full ATX supply.
It's an Aspeed AST2400 for the BMC, I can't find exact power consumption figures (other than "low power consumption" in Aspeed's marketing material) STH has an article showing that it seems to be <10 W https://www.servethehome.com/intel-avoton-rangeley-power-consumption-real-world-c2750-samples-tested/. (https://www.servethehome.com/intel-avoton-rangeley-power-consumption-real-world-c2750-samples-tested/.) However, I may just consider using a 12V wall-wart.

Quote
There's a good chance PSON can connect to the ATX signal for the same purpose, enabling powering on the server with the front panel button. But you'll need a bit more information about the board's PSON scheme to confirm (or just try it). You can probably also ignore this pin entirely and attach the ATX PSON to a separate switch and power it on that way.
Stupid question: The server's, or I guess the computer's on/off state is independent of the PSU's on/off state? As in, it's not possible for a computer to be "off" but the PSU to be on? Obviously, vice versa isn't possible. If not does this mean that if I have a separate (say external) button directly connected to the PSON Pin on the PSU that would turn on the computer with the PSU? (My understanding also is that the PSON line cycles the state when you short it, so when you press the button you are shorting it to ground?). Based on the fact you said that there's a PSON scheme my understanding is probably very wrong, but I could just try and hope it's a standard ATX PSON scheme.

Quote
I would use the solid copper core of 12 or 10 AWG house wiring available at the hardware store to parallel the pins at the connector, then solder all the power wires from one of the power supply's VGA cables to that. How much current can safely be carried is a function of acceptable temperature rise. You can find 'ampacity' charts online. The PSU wiring is probably 18AWG and there are 4 conductors for +12 and GND on one of those connectors. Should be good for 40A.

I'll probably give the house wiring a try. I don't have a soldering kit (it's been nearly a decade since I last soldered stuff in high school), seems like it's time to get one (not surprised). The PSU's VGA cables... Just to clarify I should essentially cut off the 6-pin or 8-pin ATX connector (meant for VGA cards) and then solder the 12V and Ground into the two house wires that go to the Ground and 12V, then solder the house wire onto the appropriate fingers? As for the PSU itself, the 12V rail can handle 62.5A and the 5V standby is rated at 3A, no clue about the AWG ratings on the cables and they are sheathed under a plasticy mesh.

Title: Re: Powering on Server with proprietary but documented power connector
Post by: ve7xen on January 06, 2022, 07:35:12 pm
They are 18-pin connectors that attach to the mainboard. The mainboard has 2 of these, one is left unpopulated. The node I have the left model so the powerboard is on the right and uses the mainboard's right side 18-pin. I assume the one on the left is for the right model so that Gigabyte only has to make 1 mainboard for both node types. HP has an 18-pin connector as well, but there's no guarantee that it's the same pinout. There's no EPS-like connector, so the CPUs must be pulling power from the 18-pin (along with memory).

Ahh, I got it now, I don't know why I would have thought that was an input.

Quote
It's an Aspeed AST2400 for the BMC, I can't find exact power consumption figures (other than "low power consumption" in Aspeed's marketing material) STH has an article showing that it seems to be <10 W https://www.servethehome.com/intel-avoton-rangeley-power-consumption-real-world-c2750-samples-tested/. (https://www.servethehome.com/intel-avoton-rangeley-power-consumption-real-world-c2750-samples-tested/.) However, I may just consider using a 12V wall-wart.

It's a pretty common BMC on ATX server motherboards too, so it's probably designed to fit well within the ATX-spec 15W power envelope. You'd probably be okay with a boost converter to give the board the 12V it wants from the ATX Vsb like one of the XL6009 modules widely available on eBay/Amazon/etc.

Quote
Stupid question: The server's, or I guess the computer's on/off state is independent of the PSU's on/off state? As in, it's not possible for a computer to be "off" but the PSU to be on? Obviously, vice versa isn't possible. If not does this mean that if I have a separate (say external) button directly connected to the PSON Pin on the PSU that would turn on the computer with the PSU? (My understanding also is that the PSON line cycles the state when you short it, so when you press the button you are shorting it to ground?). Based on the fact you said that there's a PSON scheme my understanding is probably very wrong, but I could just try and hope it's a standard ATX PSON scheme.

It's possible the board you have is set up differently, but in a typical ATX PC, the logic just controls power, it doesn't affect the operation of the PC. Obviously if you do it separate from the motherboard, it won't be able to turn itself off as usual, but otherwise it should work normally. More likely would be that boot depends on some kind of power good signal back from the power supply, but from the reverse engineering that other person did, that doesn't seem necessary here.

Quote
I'll probably give the house wiring a try. I don't have a soldering kit (it's been nearly a decade since I last soldered stuff in high school), seems like it's time to get one (not surprised). The PSU's VGA cables... Just to clarify I should essentially cut off the 6-pin or 8-pin ATX connector (meant for VGA cards) and then solder the 12V and Ground into the two house wires that go to the Ground and 12V, then solder the house wire onto the appropriate fingers? As for the PSU itself, the 12V rail can handle 62.5A and the 5V standby is rated at 3A, no clue about the AWG ratings on the cables and they are sheathed under a plasticy mesh.

Basically yeah. You could also buy the correct mating Molex connector for the VGA +12V while you're buying the card edge connector and build an 'adapter' instead of modifying the ATX supply's cables. Lots of ways to do it in the end, just make sure there's plenty of copper shorting all the card-edge pins that connect to each rail. You probably don't really need the house wire, just strip a good amount of two of the ATX wires, twist them and apply solder, and lay that to cover all 4/5 of the card-edge pins on one side, applying plenty of solder to make sure everything is shorted together. Do the same with the other two ATX wires on the other side, and repeat for ground, and that should be fine.
Title: Re: Powering on Server with proprietary but documented power connector
Post by: Eelt on January 07, 2022, 09:36:32 pm
Quote
They are designed to mate with card edge connectors. Looks like the board has markings A1-A14 and B1-B14, so I guess it is a 28-pin connector, probably 4mm pitch (you would be wise to measure it...). Usually rated for 3A per contact, and it looks like the 12V pad is made of either 4 or 5 contacts on both sides, so total capacity of 24A or 30A. If these assumptions are correct, the mating connector would be Hirose CR7C-28DA-4DS, I think. Mouser: https://www.mouser.com/ProductDetail/Hirose-Connector/CR7C-28DA-4DS?qs=sGAEpiMZZMvShe%252BZiYheiqTffYEOnDp8kWyRt6obIGA%3D (https://www.mouser.com/ProductDetail/Hirose-Connector/CR7C-28DA-4DS?qs=sGAEpiMZZMvShe%252BZiYheiqTffYEOnDp8kWyRt6obIGA%3D)

I just measured the power board. The board is ~3.9 cm wide, and the gold contacts end ~0.9 cm from the edge of the PCB. The datasheet that Mouser has linked for the Hirose connector doesn't seem to show the width of the connector (nor does the listing). I'm assuming pitch means the length/depth of the pin (being 0.4 cm aka 4 mm), if it means the width of the pin, then 28 * 0.4 = 11.2 cm (excluding space between pins) which would be way to wide for this board.

Quote
It's a pretty common BMC on ATX server motherboards too, so it's probably designed to fit well within the ATX-spec 15W power envelope. You'd probably be okay with a boost converter to give the board the 12V it wants from the ATX Vsb like one of the XL6009 modules widely available on eBay/Amazon/etc.
The other thing I'd like to ask about the 12Vsb rail is that with this topology there's no dedicated ground for the 12Vsb. The whole system uses one ground finger. This makes me ask the question if I use an XL6009 or a wall adapter, I'd only be able to hook up the positive/hot end? Would the XL6009 even be functional (ie have a closed circuit) since it has positive and negative terminals? I'd assume hooking up the 12V output's negative terminal to the main ground pin finger would be a very bad idea, unless the idea here would be to use a different ground pin on the ATX PSU (maybe one from the 24-pin if I'm already running a lead from the PSON pin), but even then on the output side of the board it still has a negative terminal... 
Title: Re: Powering on Server with proprietary but documented power connector
Post by: ve7xen on January 08, 2022, 12:58:35 am
I just measured the power board. The board is ~3.9 cm wide, and the gold contacts end ~0.9 cm from the edge of the PCB. The datasheet that Mouser has linked for the Hirose connector doesn't seem to show the width of the connector (nor does the listing). I'm assuming pitch means the length/depth of the pin (being 0.4 cm aka 4 mm), if it means the width of the pin, then 28 * 0.4 = 11.2 cm (excluding space between pins) which would be way to wide for this board.

Good thing you measured ;). Pitch is the centre-to-centre distance of the contacts. The dimensions are in the datasheet, I think. The nominal board width is dimension 'D' in the drawings, 57mm for the 28-pin version. It's only 14 pins per side because it's double-sided, so half as wide as you guessed, but still larger than the actual board so it must be the wrong part.

If the board is about 39mm, then I think it's probably 0.1" (2.54mm) pitch and *30* pins not 28, despite the PCB markings, just because this is a standard part. Again, you should measure and make sure. This part has a nominal board slot width of 1.6"-0.012 (40.34mm). See TE 5530843-2, Vishay EB43-S0K1540W, EDAC 345-030-520-202.

Quote
The other thing I'd like to ask about the 12Vsb rail is that with this topology there's no dedicated ground for the 12Vsb. The whole system uses one ground finger. This makes me ask the question if I use an XL6009 or a wall adapter, I'd only be able to hook up the positive/hot end? Would the XL6009 even be functional (ie have a closed circuit) since it has positive and negative terminals? I'd assume hooking up the 12V output's negative terminal to the main ground pin finger would be a very bad idea, unless the idea here would be to use a different ground pin on the ATX PSU (maybe one from the 24-pin if I'm already running a lead from the PSON pin), but even then on the output side of the board it still has a negative terminal...

You're correct that it won't work without a closed circuit. However these small modules are not isolated supplies (though they are available if you wanted that...you don't here), which means that the ground reference on both sides is the same. Vsb on the ATX side will use the same ground as the rest of the supplies, so you have ground continuity if you power the module from ATX Vsb (as long as the main power is connected too) even if you only connect the +12V output of the module. If you chose to use a wall-wart, you'd need to connect its ground to the ATX ground, though.
Title: Re: Powering on Server with proprietary but documented power connector
Post by: Eelt on January 09, 2022, 10:24:49 pm
I've decided to grab out the measuring tape again and take some photos. I also just realized that you can see the markings of the pins from the original chassis, so maybe that will help identify what card edge connector can be used. I tried my best to keep the tape flat and straight, but it's kind of hard to manage that along with a phone in your second hand...

Quote
You're correct that it won't work without a closed circuit. However these small modules are not isolated supplies (though they are available if you wanted that...you don't here), which means that the ground reference on both sides is the same. Vsb on the ATX side will use the same ground as the rest of the supplies, so you have ground continuity if you power the module from ATX Vsb (as long as the main power is connected too) even if you only connect the +12V output of the module. If you chose to use a wall-wart, you'd need to connect its ground to the ATX ground, though.
This to me sounds like the XL6009 will work with only passing 5 Vsb and no ground for input and actually get 12 Vsb out, which works! Hopefully, the BMC doesn't use that much power since from what I've seen on YouTube the XL6009 starts failing (mostly voltage drop) when a few Amps is drawn from the output but we would expect the BMC to use <0.84 A (probably closer to 0.667 A) on 12 V so I guess it's worth a shot.
Title: Re: Powering on Server with proprietary but documented power connector
Post by: ve7xen on January 10, 2022, 09:25:12 am
I've decided to grab out the measuring tape again and take some photos. I also just realized that you can see the markings of the pins from the original chassis, so maybe that will help identify what card edge connector can be used. I tried my best to keep the tape flat and straight, but it's kind of hard to manage that along with a phone in your second hand...

Using ImageJ (a handy, but a bit obtuse tool that can measure things using images with a reference scale present), I think I can confirm that this is a standard 0.1" or 2.5mm edge connector. The edge-to-pin dimensions match what's recommended too. However I'm pretty sure it has exactly 14 * 2 pins, not 15, which seems less common. TE 338030-1 looks like the right part, and Newark has it, but their description and categorization is missing/wrong, which is a bit weird. DigiKey has Sullins GBC14DCSN-S36 that looks right to me but it is much more expensive. Maybe worth a punt...I think these should work even if they're not perfect.

Quote
This to me sounds like the XL6009 will work with only passing 5 Vsb and no ground for input and actually get 12 Vsb out, which works! Hopefully, the BMC doesn't use that much power since from what I've seen on YouTube the XL6009 starts failing (mostly voltage drop) when a few Amps is drawn from the output but we would expect the BMC to use <0.84 A (probably closer to 0.667 A) on 12 V so I guess it's worth a shot.

You do still need to connect ground, but on the ATX side should be sufficient. If XL6009 isn't enough, XL4005 is a bit beefier and also has cheap modules available.
Title: Re: Powering on Server with proprietary but documented power connector
Post by: Eelt on January 11, 2022, 02:56:26 am
Quote
Using ImageJ (a handy, but a bit obtuse tool that can measure things using images with a reference scale present), I think I can confirm that this is a standard 0.1" or 2.5mm edge connector. The edge-to-pin dimensions match what's recommended too. However I'm pretty sure it has exactly 14 * 2 pins, not 15, which seems less common. TE 338030-1 looks like the right part, and Newark has it, but their description and categorization is missing/wrong, which is a bit weird. DigiKey has Sullins GBC14DCSN-S36 that looks right to me but it is much more expensive. Maybe worth a punt...I think these should work even if they're not perfect.
Cool! ImageJ sounds like a handy little tool! I went ahead and ordered a Sullins GBC14DCSN-S36 card edge connector, unfortunately, shipping costs more than the connector itself :(... I'll report back when it arrives and how it looks when connecting it to the PCB. Thanks for your help with the card edge connector!

Quote
You do still need to connect ground, but on the ATX side should be sufficient. If XL6009 isn't enough, XL4005 is a bit beefier and also has cheap modules available.

Ah, nice to know about the XL4005... As for grounding, this means that both grounds, the input and the output ground should just connect to ATX ground pins on the PSU then?