Chinese manufacturer puts hardware backdoor onto Supermicro server boards.

[BravoV]

Wonder if Supermicro lawsuit against Bloomberg on this matter (eg:defamation) can be win easily ?

[bd139]

Very easy. That would force Bloomberg to reveal their sources or pay up. This could be quite a scandal ultimately resulting in Bloomberg being accused of market manipulation.

[MK14]

You say "Lets not talk about what hasn't been proven yet. It doesn't make any sense to waste time on it."

I say "Let's talk about it until it DOES make sense, no matter how ridiculous that conversation may seem. You never know where understanding may come from."

I'm happy, for us to have differing opinions and/or ways of doing/thinking about things.
I actually think that if we agreed with each other, 100% on everything, it would probably be a bad thing.

[borjam]

There is nothing to see at the moment. Extraordinary claims require at least some evidence. It’s all words and farts.

I agree, everything is just too vague.

Windows however, and I’m quoting here “hammers the fucking shit out of the firewall even though we turn all the switches off”.

I think we’re prioritising risk vectors incorrectly here.

From incidents suffered by customers, you would be surprising at how often an asshole will use the browser on an extremely critical server (such as a NAS management controller) for that quick check of the news or the latest joke. With the additional problem that it's almost as unpatchable a machine as the typical industrial control system and due to its very nature it's extremely rare to even use AV software (not that it's a silver bullet!). The consequences are left as an exercise for the reader...

Personally I’m more worried about the nasty American monopoliser’s vampiric tendency and addiction to telemetry and activation data. Imagine the GDPR hell if some of that data contains PII one day due to a bug like the .Net core CLR telemetry logger logging command lines fully...

Let's see if GDPR really applies, the latest Facebook crap will be a good test

[Bicurico]

The whole newstory is a hoax/fake news in my opinion.

If China wants to spy on servers/computers/laptops/tablets/mobile phones, they could just put the required software hidden inside the firmware of the respective devices, for instance inside the IC managing the ethernet/mobile/wifi communication.

It beats me why someone would imagine China to solder a monitoring IC into an existing motherboard, when it could simply do it by software.

And no, doing it by software, changing the firmware and eventually even signing it again, is certainly not more difficlult than:

- developing a custom IC that is miniature for what it has to do in terms of processing power
- finding a way to connect it to the correct data lines
- finding a way to communicate with the outside world

Sorry, that simply doesn't make any sense!

Regards,
Vitor

[BravoV]

Very easy. That would force Bloomberg to reveal their sources or pay up. This could be quite a scandal ultimately resulting in Bloomberg being accused of market manipulation.

As Bloomberg is not that stupid to pull this kind of stunt.

What interesting now is to see if companies like Supermicro will take legal action, ... or maybe not at all, which is expected too. 

[borjam]

Very easy. That would force Bloomberg to reveal their sources or pay up. This could be quite a scandal ultimately resulting in Bloomberg being accused of market manipulation.

As Bloomberg is not that stupid to pull this kind of stunt.

What interesting now is to see if companies like Supermicro will take legal action, ... or maybe not at all, which is expected too. 

The thing is, with the strong denials from Apple, Supermicro and Amazon someone could be indicted of market manipulation. Either one of the former or Bloomberg. Ugly situation.

[technix]

The whole newstory is a hoax/fake news in my opinion.

If China wants to spy on servers/computers/laptops/tablets/mobile phones, they could just put the required software hidden inside the firmware of the respective devices, for instance inside the IC managing the ethernet/mobile/wifi communication.

It beats me why someone would imagine China to solder a monitoring IC into an existing motherboard, when it could simply do it by software.

And no, doing it by software, changing the firmware and eventually even signing it again, is certainly not more difficlult than:

- developing a custom IC that is miniature for what it has to do in terms of processing power
- finding a way to connect it to the correct data lines
- finding a way to communicate with the outside world

Sorry, that simply doesn't make any sense!

Regards,
Vitor

That was kind of my point. That ASPEED chip is in a convenient location for processing power (ARM9 core,) access to system RAM (over PCIe) and access to outside world (through ILO Ethernet or through injecting malware into host RAM.) It is possible that there might be a hack chip, but the hack chip won't work on its own, instead it is located on a production test pin of the ASPEED chip, injecting code into hat chip on the fly. The hack chip is literally nothing more than a microcontroller with a firmware implementing the ASPEED production test protocol and a lot of Flash space for the ARM9 payload.

[Bicurico]

These kind of chips are known in the console world as "mod chips". They will inject the appropriate data to surpass the protection mechanism.

Still, they do require a lot of computing power to "just" swap a few bits...

I could not imagine a chip as small as the one presented in the news to have enought CPU power and memory to do a useful hack based on as litte as 6(?) pins.

Also, I don't understand how they could implement that chip without having to solder any wires... It would be a miracle to have a point on the board that had the right traces on one spot where you could solder the IC.

Again, it would seem much simpler to just change the firmware or, heck, replace the whole chip they target with a hacked one.

Regards,
Vitor

[tooki]

Just because this particular flavor of industrial espionage hasn't been proven here doesn't make it NOT valid discussion. You're NOT doing anybody a service by demanding that just because it hasn't been proven here by what we can see that it is not true.

"Absence of proof is NOT proof of absence." There - scientific method. Prove it HASN'T happened. You can't, just like I can't prove it HAS happened.

Stop telling us that it isn't so, when you don't KNOW it isn't so. You BELIEVE it is not so, based on your very narrow view of the scientific method. But THAT is just as much YOUR opinion (as is your opinion of how to apply scientific method) as it is MY opinion (and that of anyone with a reasonably healthy level of cynicism) that if it isn't already happening, it will be happening tomorrow, or the next day.

It is not only probable, it is inevitable, and sooner rather than later. All you have to do is pay attention to human nature and history to know this.

THAT is where YOUR view of the scientific method differs from mine: You use it as an excuse to view the world with blinders on, while I use it to fuel my curiosity.

You say "Lets not talk about what hasn't been proven yet. It doesn't make any sense to waste time on it."

I say "Let's talk about it until it DOES make sense, no matter how ridiculous that conversation may seem. You never know where understanding may come from."

mnem
Most people, on seeing something that doesn't make sense, will pause with a dark expression on their face; be instead the person whose face brightens at the prospect.

Blah blah blah… again, nobody here has said it's impossible. We are saying it's improbable and implausible, because a) it doesn't make sense to take this approach, and b) there's no evidence that it happened as described. 

And your conspiracy theorist tone of "you have blinders on, while I'm awoke!" doesn't make you seem more enlightened, it makes you seem like, well, a classic conspiracy theorist, complete with the "I want to believe!" poster on the wall that you stole from Mulder's office.

[tooki]

Again, it would seem much simpler to just change the firmware or, heck, replace the whole chip they target with a hacked one.

To me, a much bigger piece of evidence (or rather, absence of evidence!) is that the supposedly affected companies haven't detected any suspicious traffic. Regardless of what method you use to compromise a server board, it has to be able to communicate its findings (or receive instructions) with the outside. And since such a covert chip couldn't possibly send and receive radio transmissions (through layers of metal enclosures and racks and cages) any useful distance at any useful speed, it means the data would have to flow through the NIC, and that's being monitored. Companies now routinely monitor traffic precisely to guard against attacks, so it's not as though one can just quickly send a few hundred packets unnoticed.

[Bud]

Yes but do you think Facebook did not monitor their network? Yahoo? Equifax? Data still get siphoned out in Gigabytes, not small packets, with all the possible monitoring in place.

[Cerebus]

..., so it's not as though one can just quickly send a few hundred packets unnoticed.

Be aware that in some scenarios it may only be necessary to send a little data, and quite possibly to send it in a leisurely fashion. That leads to the classic covert channel that perhaps leaks a few bits a second by random padding, packet timing, or turning on and off protocol options. If you need to leak a 128 bit key, or a 2048 bit key that's quite doable quite quickly in a covert channel, and intrusion detection systems and traffic monitoring/filtering systems generally will completely miss it. Proving that a system is free of covert channels is one of the classical hard problems of the cryptologist's world.

[CaptCrash]

Getting data out of a system via interesting means is also quite humerious/ingenious at times.

I remember an example at a partner company to where I worked where data was transmitted by issuing DNS queries from a compromised system using the DNS infrastructure as a very slow semaphore.

Whist the system had no direct internet access it did have internal DNS servers.  These internal DNS servers after a couple layers of firewalls and various other types of internal security then was able to query external DNS servers.  Data from the compromised system was collected and then transmitted by using internal DNS servers of the site these queries eventually being requested against internet DNS servers.  Unique external host names were queried (across a few domain names with GUID looking host names) with encoded data in the host names.    The data returned in the form of IP addresses allowed for control of the compromised system with some regular dns lookups every several hours.

This was a great example of not needing to transfer data in a fast manner or even directly, rather to utilize existing communications to hide in plan sight.  I'm sure this wouldn't work in all situations, but its also probably going to work in a lot.

After seeing what could be achieved in this situation, my confidence that virus/worm developers can deliver just about any functionally is fairly high, why are people skeptical that similar complexity cannot be delivered via hardware?  I understand that the level of complexity is different but so is the payoff time for doing what ever you are looking to do.

It will be interesting if in a few months Super Micro does or does not sue Bloomberg.  Surely that will answer the question one way or the other.

edited for words 

[FrankBuss]

I remember an example at a partner company to where I worked where data was transmitted by issuing DNS queries from a compromised system using the DNS infrastructure as a very slow semaphore.

How was this detected? I guess if you fully control the server, you could monitor the internet traffic and then compare all internet traffic with the installed programs. But if it is something like an Amazon cloud server, you would need to analyze every customer application. So it would be impossible to detect hidden traffic, except by detecting the hidden program itself. This makes it again more plausible to install something in the hardware, which can initiate network traffic outside of the core CPUs itself, because hidden programs with high privilege, which has suspicious network traffic, might be easier to detect. Of course, would be much better to install a modified BMC chip instead of an extra chip, maybe with 2 layers, like running the transferred firmware in the normal layer, but one hidden layer above an additional spy firmware. But would be much more expensive, if they need to change the die for it.

[bd139]

That's easy. You have private DNS, your DNS doesn't forward past the local DNS resolver and you log the NXDOMAIN responses.

All your users go via authenticated proxy (squid) or aren't on the public internet.

You can run the same in AWS. Your instances don't have to be internet facing. Just don't have an NGW on your VPC and VPN yourself into it with a VPN GW.

[mnementh]

The whole newstory is a hoax/fake news in my opinion.

If China wants to spy on servers/computers/laptops/tablets/mobile phones, they could just put the required software hidden inside the firmware of the respective devices, for instance inside the IC managing the ethernet/mobile/wifi communication.

It beats me why someone would imagine China to solder a monitoring IC into an existing motherboard, when it could simply do it by software.

And no, doing it by software, changing the firmware and eventually even signing it again, is certainly not more difficlult than:

- developing a custom IC that is miniature for what it has to do in terms of processing power
- finding a way to connect it to the correct data lines
- finding a way to communicate with the outside world

Sorry, that simply doesn't make any sense!

Regards,
Vitor

The reason you say that is because you think like a westerner, where you have to pay a third party to make the hardware. They OWN the foundries where this stuff is forged; for them electronic hardware is as fluid and dynamic as the software used to create it. It is just the CUSTOMER who has to pay for changes, because revision is their stock in trade.

A custom device, completely self-contained from the device it is monitoring, is the obvious choice from a security penetration standpoint, as EVERYTHING software that is supposed to be there has the potential to be reviewed while the device is IN USE.

And the use of such a device instantly allows deniability... it becomes much harder to track down where in the supply chain such a device was added; no way of knowing, or even guessing, whether the device was intercepted and the bug planted after the fact, or if it was contracted by one of the "Five Eyes, etc" groups to be produced in a "special run" of product that supposedly "never existed".

Really... you're thinking like a normal, sane person and attempting to apply LOGIC to the actions of government and enterprise BUREAUCRACY... that is why you can't imagine this.

Just because this particular flavor of industrial espionage hasn't been proven here doesn't make it NOT valid discussion. You're NOT doing anybody a service by demanding that just because it hasn't been proven here by what we can see that it is not true.

"Absence of proof is NOT proof of absence." There - scientific method. Prove it HASN'T happened. You can't, just like I can't prove it HAS happened.

Stop telling us that it isn't so, when you don't KNOW it isn't so. You BELIEVE it is not so, based on your very narrow view of the scientific method. But THAT is just as much YOUR opinion (as is your opinion of how to apply scientific method) as it is MY opinion (and that of anyone with a reasonably healthy level of cynicism) that if it isn't already happening, it will be happening tomorrow, or the next day.

It is not only probable, it is inevitable, and sooner rather than later. All you have to do is pay attention to human nature and history to know this.

THAT is where YOUR view of the scientific method differs from mine: You use it as an excuse to view the world with blinders on, while I use it to fuel my curiosity.

You say "Lets not talk about what hasn't been proven yet. It doesn't make any sense to waste time on it."

I say "Let's talk about it until it DOES make sense, no matter how ridiculous that conversation may seem. You never know where understanding may come from."

mnem
Most people, on seeing something that doesn't make sense, will pause with a dark expression on their face; be instead the person whose face brightens at the prospect.

Blah blah blah… again, nobody here has said it's impossible. We are saying it's improbable and implausible, because a) it doesn't make sense to take this approach, and b) there's no evidence that it happened as described. 

And your conspiracy theorist tone of "you have blinders on, while I'm awoke!" doesn't make you seem more enlightened, it makes you seem like, well, a classic conspiracy theorist, complete with the "I want to believe!" poster on the wall that you stole from Mulder's office.

"Blah, blah, blah..." you said it yourself.

Who is more the fool? The one who considers conspiracy theories and attempts to find the grain of truth behind them, or the one who cavalierly dismisses real evil, corruption and conspiracy going on all around that is so blatant it is happening right out in the open for all to see?

Calling willful ignorance "the scientific method" is just another lie, only it's the lie you tell yourself to have an excuse for that ignorance.

True "scientific method" investigates, records, and DOES NOT PRESUME ANYTHING.

It certainly does not assume that because we haven't proven a thing yet, it isn't so. It is in fact the polar OPPOSITE of that assumption.

mnem
Follow. The. Money.

[borjam]

That's easy. You have private DNS, your DNS doesn't forward past the local DNS resolver and you log the NXDOMAIN responses.

All your users go via authenticated proxy (squid) or aren't on the public internet.

You can run the same in AWS. Your instances don't have to be internet facing. Just don't have an NGW on your VPC and VPN yourself into it with a VPN GW.

You can also log DNS activity and check for unusual activity. Like:

- What are the likely domains to be contacted by a given server? For a Windows system microsoft.com makes sense, w4r3zs4nsfr0nt1eres.org.cn doesn't

- Does it contact recently created domains or have the domains been registered for a many years?

- Are the domains it contacts popular in your infrastructure or only a very small bunch of servers (or just one) query them?

- Do queries vary a lot or not?

- Where are the odd domains hosted?

- What is their registrar?

- In case there are web pages associated to those domains, what do they look like?

- Does it make a lot of queries for those odd domains?

etc etc.

[technix]

These kind of chips are known in the console world as "mod chips". They will inject the appropriate data to surpass the protection mechanism.

Still, they do require a lot of computing power to "just" swap a few bits...

I could not imagine a chip as small as the one presented in the news to have enought CPU power and memory to do a useful hack based on as litte as 6(?) pins.

Also, I don't understand how they could implement that chip without having to solder any wires... It would be a miracle to have a point on the board that had the right traces on one spot where you could solder the IC.

Again, it would seem much simpler to just change the firmware or, heck, replace the whole chip they target with a hacked one.

Regards,
Vitor

This is a lot simpler if the attacker can fab chips. An example attack:

1. The production test pin on the ASPEED chip has internal pull up to 1.2V Vcore. During normal operation this pin is pulled to ground with a 100 ohm pull-down. The pin implements some kind of one-wire serial debugging protocol.
2. The modchip is made on a 28nm node allowing for 48kB mask ROM, 2kB SRAM and a 12MHz low power 1T 8051 core. The nominal 1.2V supply plus a Schottky diode drop allows the direct use of 28nm (even 14nm) process, shrinking the die significantly allowing it to be fit on a 0603 resistor package.
3. The modchip has two terminals, DIO and GND, like those Dallas 1-wire chips commonly do. Through an internal Schottky diode and capacitor the chip can draw power and communicate through that DIO pin.
4. The firmware of the modchip implements the serial debugging protocol, capable of dumping its 32kB payload into the DRAM of the ASPEED chip and launch it.

[Gyro]

An article taking a rather more down-to-earth look at the Bloomberg motherboard hacking claim from the ElectronicDesign site...

https://www.electronicdesign.com/embedded-revolution/how-hack-server-motherboard

[thm_w]

An article taking a rather more down-to-earth look at the Bloomberg motherboard hacking claim from the ElectronicDesign site...

https://www.electronicdesign.com/embedded-revolution/how-hack-server-motherboard

There have been more details revealed lately and it appears that the motherboard circuit board did not have to be modified. Likewise, the additional chip may simply be a standard serial memory chip that was added to a location designed for the chip and left unpopulated. This is a common design approach to provide more options. For example, a TPM security chip is often an option for a server motherboard. The chip is simply left out if the motherboard will not provide that option.
...
The hack was supposedly caught, not by observing the changes to the motherboard, but by network traffic that was abnormal. A more sophisticated implementation might delay compromised communication until much later making it much harder to detect.

So the tiny filter package thing may have been completely wrong, which threw a lot of people off.

[tooki]

Just because this particular flavor of industrial espionage hasn't been proven here doesn't make it NOT valid discussion. You're NOT doing anybody a service by demanding that just because it hasn't been proven here by what we can see that it is not true.

"Absence of proof is NOT proof of absence." There - scientific method. Prove it HASN'T happened. You can't, just like I can't prove it HAS happened.

Stop telling us that it isn't so, when you don't KNOW it isn't so. You BELIEVE it is not so, based on your very narrow view of the scientific method. But THAT is just as much YOUR opinion (as is your opinion of how to apply scientific method) as it is MY opinion (and that of anyone with a reasonably healthy level of cynicism) that if it isn't already happening, it will be happening tomorrow, or the next day.

It is not only probable, it is inevitable, and sooner rather than later. All you have to do is pay attention to human nature and history to know this.

THAT is where YOUR view of the scientific method differs from mine: You use it as an excuse to view the world with blinders on, while I use it to fuel my curiosity.

You say "Lets not talk about what hasn't been proven yet. It doesn't make any sense to waste time on it."

I say "Let's talk about it until it DOES make sense, no matter how ridiculous that conversation may seem. You never know where understanding may come from."

mnem
Most people, on seeing something that doesn't make sense, will pause with a dark expression on their face; be instead the person whose face brightens at the prospect.

Blah blah blah… again, nobody here has said it's impossible. We are saying it's improbable and implausible, because a) it doesn't make sense to take this approach, and b) there's no evidence that it happened as described. 

And your conspiracy theorist tone of "you have blinders on, while I'm awoke!" doesn't make you seem more enlightened, it makes you seem like, well, a classic conspiracy theorist, complete with the "I want to believe!" poster on the wall that you stole from Mulder's office.

"Blah, blah, blah..." you said it yourself.

Who is more the fool? The one who considers conspiracy theories and attempts to find the grain of truth behind them, or the one who cavalierly dismisses real evil, corruption and conspiracy going on all around that is so blatant it is happening right out in the open for all to see?

Calling willful ignorance "the scientific method" is just another lie, only it's the lie you tell yourself to have an excuse for that ignorance.

True "scientific method" investigates, records, and DOES NOT PRESUME ANYTHING.

It certainly does not assume that because we haven't proven a thing yet, it isn't so. It is in fact the polar OPPOSITE of that assumption.

mnem
Follow. The. Money.

You did not understand my comment. You’re going off on another condescending cuckoo conspiracy theorist “I’m more aware than you!” rant/sermon, and about evil and your idea of “scientific method” and whatnot, and meanwhile you don’t even realize what I did (and didn’t) say.

No need to reply, I’ve added you to my ignore list. I don’t need the temptation of getting into arguments with conspiracy nuts.

[mnementh]

You did not understand my comment. You’re going off on another condescending cuckoo conspiracy theorist “I’m more aware than you!” rant/sermon, and about evil and your idea of “scientific method” and whatnot, and meanwhile you don’t even realize what I did (and didn’t) say.

No need to reply, I’ve added you to my ignore list. I don’t need the temptation of getting into arguments with conspiracy nuts.

No, I fully understood your comment. I called you out on it. You're the one who resorted to name calling, belittling and personal attacks; I'm just refusing to let you slide on a lie. Call it what you like, but it's still a lie.

Cheers,

mnem

[Marco]

So the tiny filter package thing may have been completely wrong, which threw a lot of people off.

Because of the nigh infinite negative number security domains we have below the OS level now, all with their own resident firmware upgrades, you don't need to do anything in the supply chain for this. If it was just a group of hackers (state sponsored or not) abusing flaws in supermicro servers it's suddenly business as usual.

IMO all firmware should be on the HD except tiny open source bootloaders, maybe with an updatable library of public keys for firmware decryption. I don't mind if the manufacturers encrypt their firmware, but stop putting ever more little bits of memory in machines which allow resident exploits (this goes for storage and extension cards as well). We have been at the point for over a decade now where you can throw an exploited machine in the fucking bin, there's too many nooks and crannies where a sophisticated actor can stuff a hook to re-rootkit you.

[bd139]

We have open firmware. It is called OpenFirmware/OpenBoot.

Unfortunately most of our infrastructure is built on a house of cards from the late 1970s with hack after hack piled on top of it (x86). This mandates a pile of drunken arse shite to get the hardware aligned with reality so the OS doesn’t vomit when it finally gets to take over from the masturbating monkey in charge of the show.

Really need to bin x86 and start again.