Japanese satellite lost due to software update

[Bryan]

Oh boy, looks someone got fired...or worse.

http://hackaday.com/2016/05/02/software-update-destroys-286-million-japanese-satellite/

[imidis]

Hmm, and I thought Windows Updates were bad. 

[Bryan]

Well, NASA probably isn't laughing. If I recall they lost a Mars orbiter in 1999 due to a imperial measurement in code when the rest of the team was using metric.

[AntiProtonBoy]

Such a shame. What setback for astronomy.

[GEuser]

Ooop's such a simple error does that , How do they do updates in Star Trek?

On a side note there might be more collateral damage? , now this "is" only speculation or could be just a glitch also , but when i downloaded the Keplerian elements a week or so ago it was filled with .DEB , deb is a short for debris , i use look around with several satellite programs and that happened to 2 (from a particular server) the other 1 uses a different server to update .

At the time i just thought it was a mix up?

[rrinker]

 Yes, they do - in the original series, season 2 episode The Ultimate Computer, they try upgrading the Enterprise's computer - and that didn't work out too well.

And in Next Generation, there is the episode where the Binars update the Holodeck computer with a "less than awesome" outcome. But then there was Minuet in that episode....with Riker going full on Kirk over her.

[macboy]

Topic is misleading, I thought this would be just another communications or spy (ahem..) military satellite but it was in fact an X-Ray telescope, and a very new and expensive one at that. This definitely deals a blow to science.

I read the JAXA failure analysis. There is no mention of software update. The Hackaday blog is simply wrong. There is mention of a center of mass update being sent from one module to another, required due to the extension of some apparatus. JAXA believes there may have been some issue with this data.

Two on-board instruments disagreed about the position and/or angular velocity of the satellite, resulting in erroneous corrective action which caused it to spin. Even when it went into a safe made, continued misunderstanding of its position or angular velocity caused additional bad corrective action which caused more spinning. In the end, the satellite physically destroyed itself by causing itself to spin at such a rate that it broke up into multiple pieces.

[GEuser]

Well i just rechecked the Sat tracking programs and updated the Keps and no DEB's are included this time in all , the last updated when they came through on 1 program was on 23/4/16 and that one uses a official server .

See , it took ages to get them out of the old program as that is the way it's just designed , for memory there was 1450 plus in total or way over that to say the least (the normal sat's and plus all the DEB's) , this current now Kep update has 1375 sats and no DEB's , i should not have got rid of them as like i mentioned i thought it was a glitch , but if they had been kept only those DEB'S could have been loaded and displayed to see how widespread it all was up there or if it made sense .

I recall there was the names of satellite that had no association too , i think there were a few "COSMOS" too ..? , they (the kep) started with the original satellite name then listed in order of the same name progressing in number all named at the end line DEB , then the next sat and so on and so on .

But still maybe just a coincidence or a bug .

edited some spelting .

[MrSlack]

This doesn't surprise me. I dont like making stereotypes but I can't avoid it here. The Japanese have always been really good at the hardware. Unfortunately software is always marked with resignations, apologies, facepalms and frustration and I'm 100% sure it's a cultural issue. I've worked with two disparate Japanese software teams and they both hid every fuck up under a pile of excuse and pointed further up the hierarchy until someone resigned or wrote an extensive apology. Same with a colleague who worked in the gaming sector on tens of projects. Same with Sony on numerous occasions.

[Vgkid]

D'oh 

[MrSlack]

They have insurance for these sorts of things I think so the only thing is the schedule and the premium.

[coppice]

They have insurance for these sorts of things I think so the only thing is the schedule and the premium.

Commercial satellites are generally insured. Government funded research craft are not.

[dannyf]

That shows you how important it is to design against failures. I still curse today at a BMW embedded engineerer who wrote a piece of shit window motor controller that couldn't be overriden by the user.

Those windows have a sensor in the upper window rubber strip that senses a jammed finger (or worse) and retards the motor automatically.

For me, that malfunction happened on a day with rain pouring out of the sky and ruined a jacket that I really liked.

The same thing here. People who wrote the software never considered cases where some devices failed to work right.

[PA0PBZ]

Two on-board instruments disagreed about the position and/or angular velocity of the satellite

When that happened it should have called home and wait for an answer. Did they really not think about that or even worse, think it could never happen?

[Red Squirrel]

Time to send Jebediah Kerman up to do a system restore manually. 

Actually is it a thing to send a space craft to a satellite to do maintenance on it?   Or is it easier/cheaper to just build a new one and send it up and consider the old one as space junk?

Edit: missed the part about it actually gotten physically damaged too from the spin, yeah I guess this is definitely a write off now.

[rich]

Isn't that most of the cost of a satellite, aside from launching service fee, come from just R&D cost? If so, building an exact replicate and send it to the space won't cost that much, right?

Probably best to avoid an exact replica of the self-destructing version, so more R&D spend required before sending v2.0 up.

[CJay]

Time to send Jebediah Kerman up to do a system restore manually. 

Actually is it a thing to send a space craft to a satellite to do maintenance on it?   Or is it easier/cheaper to just build a new one and send it up and consider the old one as space junk?

Yes, it was a thing, Hubble probably being the most high profile case but I think it's been done for several others as well.

Of course now there's no Shuttle program who knows...

Edit: missed the part about it actually gotten physically damaged too from the spin, yeah I guess this is definitely a write off now.

[station240]

Well back to the old drawing board.
I expect it will be redesigned with multiple redundant sensors, perhaps a 3 axis G sensor would make a more suitable "known" working sensor for detecting if the ship is rolling.

If you read the report, it has a timeline which shows the sat completed one observation task, and was changing position to do the next.
Also says two of the eleven pieces of debris are due to burn up in reentry soonish.

Shame to see such a valuable piece of instrumentation turned into junk like this.

[VK3DRB]

Bad software engineering.

Reminds me of a case in Sydney where a taxi company did a software update to taxi radios over the air which killed about 2000 taxi radios. And no further software update would work. Every taxi had to return to a base to have the radios removed and reprogrammed.

[kfitch42]

I read the JAXA failure analysis. There is no mention of software update. The Hackaday blog is simply wrong. There is mention of a center of mass update being sent from one module to another, required due to the extension of some apparatus. JAXA believes there may have been some issue with this data.

Further down in the hackaday article is says:

The satellite configuration information uploaded earlier was wrong and the reaction wheels made the spin worse.

I think the "configuration information" is the "software update" that hackaday is referencing. It sounds like this would have been a recoverable failure if that "configuration information" was correct.

I think hackaday was referring to this line from the JAXA presentation they linked to:

Through the post-incident investigation, it is confirmed the RCS control parameter on Feb 28th was not appropriate. This indicates
the possibility of insufficient verification of the process from creating the parameter to setting it on the satellite. This situation is
under investigation.

[kfitch42]

Two on-board instruments disagreed about the position and/or angular velocity of the satellite

When that happened it should have called home and wait for an answer. Did they really not think about that or even worse, think it could never happen?

The problem is that attitude control is a prerequisite for calling home and/or waiting for an answer. Generally these guys can't communicate so well when rotating. They need the antenna pointed toward earth. In fact the switchover to "safe-hold" was the satellite's reaction to what it thought was a broken reaction-wheel. Basically "safe-hold" was their way of dealing with the unexpected so that they could call home and/or wait for new instructions.  Unfortunately "safe-hold" only made things worse because of a bad "RCS control parameter."

[fanOfeeDIY]

This doesn't surprise me. I dont like making stereotypes but I can't avoid it here. The Japanese have always been really good at the hardware. Unfortunately software is always marked with resignations, apologies, facepalms and frustration and I'm 100% sure it's a cultural issue. I've worked with two disparate Japanese software teams and they both hid every fuck up under a pile of excuse and pointed further up the hierarchy until someone resigned or wrote an extensive apology. Same with a colleague who worked in the gaming sector on tens of projects. Same with Sony on numerous occasions.

I am Japanese and was working for Japanese manufacturing companies, and your comments is pretty true, Unfortunately.
It is more as an organization issue, it is hurting me deeply because I am making my living in software engineering field.
The organization issue is difficult to fix and probably it is not going to change or improve for a while.

Good side is that hardware engineers will get paid better than software engineer.

[MrSlack]

I noticed it doesn't seem as prevalent in the younger companies. Perhaps it's an issue with older generations. We don't respect them as much in the UK

[jitter]

I'm 100% sure it's a cultural issue.

I am Japanese and was working for Japanese manufacturing companies, and your comments is pretty true, Unfortunately.
It is more as an organization issue, it is hurting me deeply because I am making my living in software engineering field.
The organization issue is difficult to fix and probably it is not going to change or improve for a while.

I was wondering about that too and remembered a documentary on the Amagasaki rail crash. The investigation blamed a cultural phenomenon called "nikkin kyoiku" (re-education Japanese style), and I was wondering if this work ethic is common among software engineers too, or in all kinds of organizations.

I noticed it doesn't seem as prevalent in the younger companies. Perhaps it's an issue with older generations. We don't respect them as much in the UK

Perhaps younger generations are more open to other styles of work ethics.

[fanOfeeDIY]

Isn't that most of the cost of a satellite, aside from launching service fee, come from just R&D cost? If so, building an exact replicate and send it to the space won't cost that much, right?

This is the problem.
The plan for the next x-ray astronomy satellite is planned only on 2028 with "Athena", there are no replacement for ASTRO-H since the previous x-ray satellite is already retired.

The ASTRO-H was the collaboration with Japan, US, Canada, and Europe, and it has the NASA's x-ray sensors.
So it is difficult to make a quick replacement while all the countries suffers with budgets.
http://phys.org/news/2016-04-x-ray-astronomers-anxious-good-news.html

Yes, x-ray astronomy satellite is only for science purpose only and does not directly improve our daily livings but big lost for our science that have to wait nearly 10 years.
Often the technology found by pure science interest indirectly made our current living.

It is sad result.