95 kg drone with howyadoin' Arduino bodgery crashes - made in Straya!

[Syntax Error]

@eevblog, Dave, could you get a video or something out of the concept of failsafe systems?

Staggering their power cut relay was always closed. Relying on the loss of a LoRa (how far does that transmit?) signal to then energise the relay, seems like a design dreamed up by someone with an Arduino breadboard starter kit. Hang on a moment, their kill switch subsystem IS built on a breadboard-to-solder board?!

This said, I am sure there are plenty of EEs here who have experienced many 'failsafe on' designs.

From the AAIB report:

[bd139]

This was interesting.

Just them looking at their own shit show and still being of the mindset "yeah we'll fly it anyway" shows the perpetrators should have an example made out of them. This is the same mindset of criminals; they chose to put people in danger intentionally through selfishness and apathy and lied about it. There are no excuses.

Now I'm going into kill mode because I'm quite frankly fucking fed up of some of the stuff I am seeing these days coming from the "maker community". All I see is deathtraps, fire hazards, shit falling out of the sky and people blindly copying this as the status quo. What is even worse is the ceremonious back patting that comes with it, the trite cuteness and general drenching of marketing ooze. It's quite frankly embarrassing as a human looking at some of this shit and people thinking it's ok to kick out there. It really fucking pisses me off.

I've had my own run in with this shit...

I'm in the software industry and we tend to have similar non-engineering background folk coming up with all sorts of random shit all the time. In this case it was some "build traffic lights". So when the main software build fails it goes red, when it's ok it goes green so you can be shamed into fixing it. Well off goes the self-appointed maker and buys a 240V traffic lights head from ebay. This is immediately equipped with an arduino and a relay board, the only tools being employed being a swiss army knife, a couple of screwdrivers and some double sided tape. It appeared when I was away for a week I suspect because he knew I was going to rip it a new arsehole. Well I did. The entire thing was metal chassis, not even earthed - he'd cut the earth wire off. The mains lead was tied in a knot behind the sharp edges of the chassis which had already started cutting through it. All the boards were double sided taped inside and the thing was wired up with 7/0.2 wire which the insulation had already partially melted away. This was left plugged into a random desktop PC in the corner and hung off a coat hook on the coat rail where it was bashed around a people went in and out of the office. When I arrived back I took it and threw it in the skip. When I shook it, the relay board came unstuck. It probably would have with the heat anyway and killed whoever touched it next.

The frustrating bit? I got complained at for the attitude. I just saved you from burning your office to the ground or electrocuting your staff. Fuckwits.

And that's the problem: fuckwits. Wall to wall fuckwits. Need to lock their hands away somewhere separate from the rest of them.

Edit: apologies for the tone - I had a crappy day fixing half arsed engineering...

[AVGresponding]

@eevblog, Dave, could you get a video or something out of the concept of failsafe systems?

Staggering their power cut relay was always closed. Relying on the loss of a LoRa (how far does that transmit?) signal to then energise the relay, seems like a design dreamed up by someone with an Arduino breadboard starter kit. Hang on a moment, their kill switch subsystem IS built on a breadboard-to-solder board?!

This said, I am sure there are plenty of EEs here who have experienced many 'failsafe on' designs.

Yes, and it's really quite staggering how people that apparently manage to tie their own shoelaces of a morning can't understand what the word "failsafe" means.

[bd139]

Problem is it’s not how people conceptualise things. The general way people think is assuming success and looking out for danger. But that doesn’t work for unexpected scenarios. So you have to assume failure and know what criteria and measurements you can make to maintain success.

[AVGresponding]

I prefer to expect failure, and think how that can be made to happen safely, then think of ways to prevent the failure.
I guess that makes me a pessimist!   

[Syntax Error]

I'm in the software industry and we tend to have similar non-engineering background folk coming up with all sorts of random shit all the time. In this case it was some "build traffic lights". So when the main software build fails it goes red, when it's ok it goes green so you can be shamed into fixing it.

Do you work in Shordich or Kings Cross London? Google has turned the whole tech workplace into a dissolute campus playground for cash-rich skill-short startups. Oh joy, another nerd with a nerf gun.

I guess that makes me a pessimist!

Only if you're holding that half empty glass upside down.
I looked up those modules used in their 'avionics bay' (the grey plastic box used for garden fountains). The usual hobbist fare: A £4 Arduino Nano compatible (ATmega328P 16MHz), a £24 LoRa 433Mhz telemetry module, a £3 single channel 5V relay module and, a £3 GY-521 MPU-6050 3-axis 16-bit gyroscope-accelerometer module. In all, £100 worth of modules and parts controlling the whole UAV's guidance system.

The GY-521 MPU-6050 accelerometer module, of which there were two pictured, is an I2C (only) bus device. Thinks, an I2C bus monitoring the pitch-yaw-roll of a 95 kilo flying lawn mower. What could possibly go wrong?

The GY-521 datasheet suggests applications for this sensor include, "location based services and dead reckoning". I reckon dead too.
http://www.haoyuelectronics.com/Attachment/GY-521/mpu6050.pdf

_{Anyone seen my drone?}

[bd139]

So basically the avionics package was about 40 quid?

Do you work in Shordich or Kings Cross London? Google has turned the whole tech workplace into a dissolute campus playground for cash-rich skill-short startups. Oh joy, another nerd with a nerf gun.

I did bounce around the startup scene there for many years but it was limited to ripping them off as a contractor. My house is built on the crushed dreams of millennial hipsters.

[tszaboo]

Problem is it’s not how people conceptualise things. The general way people think is assuming success and looking out for danger. But that doesn’t work for unexpected scenarios. So you have to assume failure and know what criteria and measurements you can make to maintain success.

You know, there is something I like about the job I'm doing now. I have to follow a standard. This standard is saying:

Lets assume that every single component in your system can fail, there are multiple ways for this.
Prove, that it is safe, even after the failure.

[Cerebus]

I know my opinion may/will be criticised, but I did read through a big chunk of the report and I don't think this project was nearly as bad as some comments on this forum may suggest.

The pilot did comply with all/most norms, as far as I can tell. When I saw the post starting this thread, i immediatly imagined a home-made UAV operated by an amateur pilot without any authorisation, which was definitely not the case.

You obviously didn't read enough of the report to be aware of these things:

The approval specified, as a condition, where the operator was to be on the day, they were positioned somewhere else.
The approval specified, as a condition, where the observers were to be on the day, they were positioned somewhere else.
The approval specified, as a condition, the operating area, the UAV was flown outside of it.
The approval specified, as a condition, the take off point, the take off was from somewhere completely different.
The approval was them using a transmitter operating at 25mW, a 10mW transmitter was used.

Basically the operators departed wholesale from the operating procedures agreed for the day. There's more, but that alone is an indication that the operators were woefully negligent in complying with their obligations.

In light of that the CAA has let them off very lightly, I suspect in embarrassment at their own failures in not having adequate procedures or staff who had a clue how to go about managing the whole process - it seems the department involved has been put together in the most ramshackle fashion possible (see the CAA report for details).

I recommend reading the whole report. It's quicker and easier to read than its page count might at first suggest.

[mikeselectricstuff]

I'm quite frankly fucking fed up of some of the stuff I am seeing these days coming from the "maker community".
I've had my own run in with this shit...
....
..... buys a 240V traffic lights head from ebay. This is immediately equipped with an arduino and a relay board, the only tools being employed being a swiss army knife, a couple of screwdrivers and some double sided tape.....The entire thing was metal chassis, not even earthed ..... All the boards were double sided taped inside and the thing was wired up with 7/0.2 wire which the insulation had already partially melted away.

I can totally relate - I've lost count of the times I've come in to a "design studio" of some sort who realised too late that "Arduino does not scale" and need me to tear it up and build something from scratch in a week.

Typical example being someone who thinks a "bench power supply" is a Mean Well (or cheaper) box with maybe half the strands of a mains cable connected to the exposed screw terminals, which has become buried under a few sheets of paper on an office desk just waiting to bite the person sitting opposite, onto whose desk it had encroached.

 Or someone whose sole means of disconnecting a large sprawling mains powered setup is to pull out the plug from the extension lead lying on the floor...

Or someone who, during an expensive session in a film studio can't figure out why his 12V lighting setup doesn't work when it's run through 25 metres of thin speaker cable...

I could go on...

I'm truly amazed I've not heard of serious injuries or fire incidents - maybe they were embarrassed to mention.

[coppice]

I can totally relate - I've lost count of the times I've come in to a "design studio" of some sort who realised too late that "Arduino does not scale" and need me to tear it up and build something from scratch in a week.

"Arduino would not pass even the most basic EMC test" is also a big issue here. 95 kilos of electrically noisy motor systems driven by an Arduino is a death trap.

[Cerebus]

And that's the problem: fuckwits. Wall to wall fuckwits. Need to lock their hands away somewhere separate from the rest of them.

The problem with that solution is that to lock their hands up somewhere separate from them you first need the hands to be separated. Leave them alone for an hour in a workshop with some stock, a Bandsaw, a Lathe, a Milling machine and a copy of "A Makers Guide to Metalwork" ought to get the job done efficiently and with the minimum of human involvement. 

I'm so with BD139 on the walking personifications of the Dunning–Kruger effect that perpetrate this crap. The most dangerous thing about these people, and the cult of personality that goes with it, is that when challenged by people who actually know what they are talking about they don't react the way any intelligent person would and reevaluate their level of understanding, they just go into denial.


For anyone who hasn't yet realised what a shitshow this was, here's one resistor from what was, let us remember, a safety critical system.



There is no way that is acceptable in a safety critical system, with no redundancy, that flies 95 kg around in the air and that was capable of getting high enough to return to earth at terminal velocity with a kinetic energy of at least 100 kJ.

There are more pictures in the report of egregiously bad construction. All the electronics in the UAV were held in place with bloody nylon cable ties just looped about the frame members - not a screw, shakeproof washer or safety wire anywhere in sight.

The CAA don't come out of this covered in glory either. From the report:

The CAA’s UAS Unit consists of two sections, the Policy Team and the Sector Team. The Sector Team has responsibility for the oversight and management of OSC’s. At the time of the application, the CAA’s UAS Sector Team comprised of a Section Lead, one UAS Technical Inspector and two UAS Technical Surveyors. There were plans to recruit a further two Inspectors and two Surveyors. The Section Lead, who was the signatory on the exemption, had joined the CAA in May 2018 from an emergency services organisation where they had introduced UAS operations; he has since left the CAA. Other members of the UAS Sector Team joined the CAA from university and initially worked in data entry roles within the CAA’s Shared Services Centre. The Technical Inspector took up the role in January 2019 after working as a UAS Technical Surveyor for approximately one year. One UAS Technical Surveyor, was still undergoing initial training and development.

Translation: The section leader had organised drones for the police or fire brigade and probably had no prior regulatory experience, or aerospace experience. Most, possibly all, of their staff all came from university straight into the CAA to work as data entry clerks and they decided it would be a good idea to put them to work as "Technical Inspectors" or "Surveyors". You can guess what's happened by what is omitted, all mentions of prior qualifications or experience. For all we know they were English or Philosophy graduates and the fact that they don't call out degrees in relevant areas is a strong indicatior to me that I'm on the right path.

Edit: Re-reading this I'm reminded of a phrase that commonly crops up in UK criminal that goes something like this: "A personal will be guilty of an offence if they knew or should have known that <something they omitted to do was criminally negligent>". You'll find that as far back as Victorian laws. There's a duty on people to recognise when they are out of their depth so that if they didn't know enough to know whether something was right or wrong they should find someone who does know what is right and what is wrong. That bloody resistor screams at me that somebody "should have known" that it's wrong or should have had the sense to "ask a grown up".

[Cerebus]

I can totally relate - I've lost count of the times I've come in to a "design studio" of some sort who realised too late that "Arduino does not scale" and need me to tear it up and build something from scratch in a week.

"Arduino would not pass even the most basic EMC test" is also a big issue here. 95 kilos of electrically noisy motor systems driven by an Arduino is a death trap.

The Arduinos were driving ESCs which were in turn driving the motors. That is no comfort though. A lot of those ESCs are designed and built by drone hobbyists who've 'gone pro'. They graduate to circuit design, but globbing together bits of sub-circuit and fiddling with values until it all sort-of works. We had a relatively recent discussion where it became clear that the denizens of that community have little clue how it all really works. I watched a few videos and the lack of basic technical grasp of how motors and motor control systems work is astounding. One of the 'truths' circulating in the hobby drone community is that if you run a BLDC motor with no load (i.e. propellor) it runs away faster and faster until it explodes. No amount of demonstrating to one such hobbyist that this was not true because BLDC motors are only capable of running as fast as the driving current waveform would be accepted. They insisted on clinging to their beliefs even though there was patiently explained hard evidence in front of them. The probable explanation for the belief is that the ESCs are badly designed and the controllers can exhibit this runaway behaviour, but the cargo cult explanation is that "BLDC motors cannot be safely run unloaded.".

Faced with that level of lack of basic technical knowledge or information, and unwillingness to learn, preferring to follow some sort of 'Maker' cargo cult, how the hell are you going to get them to understand and mitigate EMC issues? You'll get back something like "Well I've been doing this for X years, and it works. So what I know must be right, and if this 'EMC' thing you're talking about is a problem I would have encountered it, and I haven't, so it isn't.".

The next week an EMC problem you warned them about sends their drone at full speed into the queue of little kids waiting for an ice cream at the county fair where they are doing a demonstration flight, and the local CB'ers are stoking up their linears on a stall the other side of the field.

[SilverSolder]

If these people had been car hobbyists instead of into drones/electronics, there would have been higher requirements that they would have to pass in most countries / states!

Certainly around here, if you made your own car, it would have to be inspected in order to be licensed to be on public roads.  And the inspectors know what they are doing...

[vk6zgo]

@eevblog, Dave, could you get a video or something out of the concept of failsafe systems?

Staggering their power cut relay was always closed. Relying on the loss of a LoRa (how far does that transmit?) signal to then energise the relay, seems like a design dreamed up by someone with an Arduino breadboard starter kit. Hang on a moment, their kill switch subsystem IS built on a breadboard-to-solder board?!

This said, I am sure there are plenty of EEs here who have experienced many 'failsafe on' designs.

Yes, and it's really quite staggering how people that apparently manage to tie their own shoelaces of a morning can't understand what the word "failsafe" means.

Yeah!
Before it had a fancy name, people who built Radio Transmitters with serious "biteys" available inside, knew that the default position of a safety related relay or contactor was "OFF", & anything that could otherwise allow personnel contact, or fire, was "safety related".

[beanflying]

If these people had been car hobbyists instead of into drones/electronics, there would have been higher requirements that they would have to pass in most countries / states!

Certainly around here, if you made your own car, it would have to be inspected in order to be licensed to be on public roads.  And the inspectors know what they are doing...

There is but the point is they lied and the UK Authorities didn't do their job of inspection to their rules.

@eevblog, Dave, could you get a video or something out of the concept of failsafe systems?

Staggering their power cut relay was always closed. Relying on the loss of a LoRa (how far does that transmit?) signal to then energise the relay, seems like a design dreamed up by someone with an Arduino breadboard starter kit. Hang on a moment, their kill switch subsystem IS built on a breadboard-to-solder board?!

This said, I am sure there are plenty of EEs here who have experienced many 'failsafe on' designs.

From the AAIB report:

Apart from one Drone as far as I know Dave has had no real background in R/C or UAV use or building. This is is a large part of the failings of this abomination as it was put together in a rush by people without background or relevant experience. Having a specific what would be a largely speculative video redesigning a safety system for part of a larger unknown one would be of little use without this background.

Even in off the shelf R/C level gear most have a throttle off functionality on loss of signal built in and a lot of them have had this for a few decades that shut the Engine throttle servo. This was generally to slow down the likely crash to come but I have seen up way to close a 3kg Helicopter going into this shutdown to then reboot and plow in at full throttle 10m away. This was the result of a fairly early Spread Spectrum receiver having a reboot time of nearly 1 second after a number of dropped frames was detected. 1 second at 120 km/hr (estimated at the time) is about 30m add to that any chance of a recovery by the pilot and you are twice to three times that. Two of us got lucky that day and we were at the time the 30m back from the flight line as per the Oz Regulations

This 95kg UAV capable of 160km/hr (their claim) is capable of covering 40+ m/sec even when shutdown as a straight ballistic projectile. If it is at altitude that potential distance travelled in this state will be greater so a shutdown is NOT an appropriate safety in particular when being used near an Audience. At a minimum a ballistic parachute should have been required as part of any failsafe system.

[vk6zgo]

I can totally relate - I've lost count of the times I've come in to a "design studio" of some sort who realised too late that "Arduino does not scale" and need me to tear it up and build something from scratch in a week.

"Arduino would not pass even the most basic EMC test" is also a big issue here. 95 kilos of electrically noisy motor systems driven by an Arduino is a death trap.

The Arduinos were driving ESCs which were in turn driving the motors. That is no comfort though. A lot of those ESCs are designed and built by drone hobbyists who've 'gone pro'. They graduate to circuit design, but globbing together bits of sub-circuit and fiddling with values until it all sort-of works. We had a relatively recent discussion where it became clear that the denizens of that community have little clue how it all really works. I watched a few videos and the lack of basic technical grasp of how motors and motor control systems work is astounding. One of the 'truths' circulating in the hobby drone community is that if you run a BLDC motor with no load (i.e. propellor) it runs away faster and faster until it explodes. No amount of demonstrating to one such hobbyist that this was not true because BLDC motors are only capable of running as fast as the driving current waveform would be accepted. They insisted on clinging to their beliefs even though there was patiently explained hard evidence in front of them. The probable explanation for the belief is that the ESCs are badly designed and the controllers can exhibit this runaway behaviour, but the cargo cult explanation is that "BLDC motors cannot be safely run unloaded.".

It could also be a garbled version of the true fact that ordinary series DC/universal motors can suffer destructive runaway, if operated at full voltage without load.
Apart from us being taught that, I remember a series motor flying apart, with bits zipping past my ears, when our Tech lecturer forgot to turn the variac down prior to demonstrating one of them to the class.

I understand that BLDC motors are different, but they won't understand!

Faced with that level of lack of basic technical knowledge or information, and unwillingness to learn, preferring to follow some sort of 'Maker' cargo cult, how the hell are you going to get them to understand and mitigate EMC issues? You'll get back something like "Well I've been doing this for X years, and it works. So what I know must be right, and if this 'EMC' thing you're talking about is a problem I would have encountered it, and I haven't, so it isn't.".

The next week an EMC problem you warned them about sends their drone at full speed into the queue of little kids waiting for an ice cream at the county fair where they are doing a demonstration flight, and the local CB'ers are stoking up their linears on a stall the other side of the field.

[Kean]

So basically the avionics package was about 40 quid?

Maybe that is all they could afford after spending most of their budget on the airframe.  The airframe that is now a pile of scrap metal (or scrap carbon fibre?).   

[Syntax Error]

Apart from one Drone as far as I know Dave has had no real background in R/C or UAV use or building. This is is a large part of the failings of this abomination as it was put together in a rush by people without background or relevant experience. Having a specific what would be a largely speculative video redesigning a safety system for part of a larger unknown one would be of little use without this background.

Actually, I was not specifically thinking of failsafe devices in UAVs, but the wider implication of failsafe-by-design in all electronic engineering.

As NANDBlog aptly says:

Lets assume that every single component in your system can fail, there are multiple ways for this.
Prove, that it is safe, even after the failure.

[Via Google] the IEEE has a working group on a " Standard for Fail-Safe Design of Autonomous and Semi-Autonomous Systems"

This standard establishes a practical, technical baseline of specific methodologies and tools for the development, implementation, and use of effective fail-safe mechanisms in autonomous and semi-autonomous systems. The standard includes (but is not limited to): clear procedures for measuring, testing, and certifying a system's ability to fail safely on a scale from weak to strong, and instructions for improvement in the case of unsatisfactory performance. The standard serves as the basis for developers, as well as users and regulators, to design fail-safe mechanisms in a robust, transparent, and accountable manner.

https://standards.ieee.org/project/7009.html

[bd139]

Hmm standards...

1. Experts get together and define a working group.
2. Experts eat a lot of biscuits and drink a lot of tea.
3. Eventually out pops a standard revision.
4. Consultancies align around that standard and certification of it.
5. People hire the consultancies to fix their product lifecycle and certify it.
6. Lots of paperwork gets filled in.
7. Some very small process changes are made which allow the existing processes to continue almost unaltered.
8. Product falls out of sky anyway
9. Vendor points to consultants.
10. Consultants point to IEEE.
11. Regulator shrugs
12. GOTO 1

Sorry just got years of being burned by ISO crap which turned into a way of hiring from rock bottom in the pool and paying consultants to certify away all the gaps and incompetence. And don't tell me that doesn't happen in aviation - look at Boeing.

Only thing that actually improves standards is hiring clue and body counts unfortunately. Every safety process has a death behind it because objectively no one gives a fuck until they have to shovel a corpse away.

Edit: I suppose this is why aviation has higher standards than general software actually. One has a body count and one does not.

[janoc]

So basically the avionics package was about 40 quid?

Maybe that is all they could afford after spending most of their budget on the airframe.  The airframe that is now a pile of scrap metal (or scrap carbon fibre?).   

That looks more like someone who knows how to build model aircraft/multicopters realizing that they also need a control system and the stock RC airplane/drone fare wouldn't do for a vehicle of this size.

"Oh but I have a nephew who is really good with Arduinos!" - so the dude came, ordered a bunch of cheap parts from AliExpress and made it "work". Given the construction it was actually a pretty remarkable achievement that it worked as well as it did.

Fortunately nobody has died.

[coppice]

Hmm standards...

1. Experts get together and define a working group.
2. Experts eat a lot of biscuits and drink a lot of tea.
3. Eventually out pops a standard revision.
4. Consultancies align around that standard and certification of it.
5. People hire the consultancies to fix their product lifecycle and certify it.
6. Lots of paperwork gets filled in.
7. Some very small process changes are made which allow the existing processes to continue almost unaltered.
8. Product falls out of sky anyway
9. Vendor points to consultants.
10. Consultants point to IEEE.
11. Regulator shrugs
12. GOTO 1

Sorry just got years of being burned by ISO crap which turned into a way of hiring from rock bottom in the pool and paying consultants to certify away all the gaps and incompetence. And don't tell me that doesn't happen in aviation - look at Boeing.

Only thing that actually improves standards is hiring clue and body counts unfortunately. Every safety process has a death behind it because objectively no one gives a fuck until they have to shovel a corpse away.

Edit: I suppose this is why aviation has higher standards than general software actually. One has a body count and one does not.

That's a very British view of standards committees. You need to get out more. In some places they use a radically different approach where they substitute coffee for tea, and cookies for biscuits.

[bd139]

That’s fair. I worked for a US defence company. We had donuts and coffee 

[coppice]

That’s fair. I worked for a US defence company. We had donuts and coffee 

You must have started late. The early starters get blueberry muffins.

[Cerebus]

Hmm standards...

1. Experts get together and define a working group.
2. Experts eat a lot of biscuits and drink a lot of tea.
3. Eventually out pops a standard revision.
4. Consultancies align around that standard and certification of it.
5. People hire the consultancies to fix their product lifecycle and certify it.
6. Lots of paperwork gets filled in.
7. Some very small process changes are made which allow the existing processes to continue almost unaltered.
8. Product falls out of sky anyway
9. Vendor points to consultants.
10. Consultants point to IEEE.
11. Regulator shrugs
12. GOTO 1

Sorry just got years of being burned by ISO crap which turned into a way of hiring from rock bottom in the pool and paying consultants to certify away all the gaps and incompetence. And don't tell me that doesn't happen in aviation - look at Boeing.

Only thing that actually improves standards is hiring clue and body counts unfortunately. Every safety process has a death behind it because objectively no one gives a fuck until they have to shovel a corpse away.

Edit: I suppose this is why aviation has higher standards than general software actually. One has a body count and one does not.

Also the people who end up going around all the standards committees and meetings are often effectively seconded full-time to them because they're the fools who are a pain-in-the-arse and/or too dangerous to have around the office/lab but aren't actually bad enough that you can fire them. It makes them feel important and it keeps them out from under your feet.

Weirdly the organizations that subscribe to this practice of offloading their problem staff into the standards treadmill are often the ones that are most enthusiastic about using the same standards that they know have been constructed by a committee of offloaded problem staff. Go figure!