Author Topic: Planting Undetectable Backdoors in Machine Learning Models  (Read 821 times)

0 Members and 1 Guest are viewing this topic.

Online SiliconWizardTopic starter

  • Super Contributor
  • ***
  • Posts: 15800
  • Country: fr
 
The following users thanked this post: RoGeorge

Offline TomKatt

  • Frequent Contributor
  • **
  • Posts: 529
  • Country: us
Re: Planting Undetectable Backdoors in Machine Learning Models
« Reply #1 on: March 01, 2023, 12:51:40 pm »
These AI type applications have become so complex, with datasets so large, that human minds can no longer understand their operation.  And if you don't understand how something works, how would you ever really be able to identify errors or malicious activity?

Quote from: Big Think
The computers that run those services have programmed themselves, and they have done it in ways we cannot understand. Even the engineers who build these apps cannot fully explain their behavior.

https://bigthink.com/the-future/black-box-ai/
Several Species of Small Furry Animals Gathered Together in a Cave and Grooving with a PICt
 

Offline tszaboo

  • Super Contributor
  • ***
  • Posts: 8218
  • Country: nl
  • Current job: ATEX product design
Re: Planting Undetectable Backdoors in Machine Learning Models
« Reply #2 on: March 01, 2023, 12:59:36 pm »
So what is exactly the danger with this?
As I understand this it would work the following: Having two models, one doing the expected operations, one the backdoor. So for example if you pass an image of a "blue stopsign next to a road" to an image classifier, object detector, it will tell you it's a cat. How can they exploit this?
 

Offline tom66

  • Super Contributor
  • ***
  • Posts: 7336
  • Country: gb
  • Electronics Hobbyist & FPGA/Embedded Systems EE
Re: Planting Undetectable Backdoors in Machine Learning Models
« Reply #3 on: March 01, 2023, 03:01:03 pm »
These AI type applications have become so complex, with datasets so large, that human minds can no longer understand their operation.  And if you don't understand how something works, how would you ever really be able to identify errors or malicious activity?

This is the same issue with autonomous vehicles.  The prediction by some was that we could build a neural network that takes an image in, runs it through the network, and generates steering, accelerator, brake, etc instructions.   Unfortunately, that's effectively impossible to prove safe.   The reality is that every successful autonomous vehicle out there uses the NN to process images, and solve the path and free space problem, but ultimately that data is processed by an ordinary algorithm running on an ordinary CPU, with that algorithm having been created by engineers sitting at their desks.
 

Offline AndyC_772

  • Super Contributor
  • ***
  • Posts: 4315
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Re: Planting Undetectable Backdoors in Machine Learning Models
« Reply #4 on: March 01, 2023, 03:18:24 pm »
The example they give is about a hypothetical NN used to assess creditworthiness and approve or deny bank loans.

The 'clean' NN would make a reasonable attempt to decide whether or not someone should be offered credit, based on the information available to the bank at the time.

The 'backdoored' NN would, however, be made to unconditionally approve a loan, if some very particular, carefully crafted conditions were met. For example, a loan of $10,000 might be rejected, but a loan to the same individual of precisely $10,004.26 would be approved.

Crucially, the agency that performed the training would be able to know what criteria would result in these 'false' approvals, but it would be mathematically very difficult to detect that they existed without that inside knowledge.

Offline daqq

  • Super Contributor
  • ***
  • Posts: 2321
  • Country: sk
    • My site
Re: Planting Undetectable Backdoors in Machine Learning Models
« Reply #5 on: March 01, 2023, 10:20:13 pm »
So what is exactly the danger with this?
Of the top of my head:
You use a neural network to recognize bank notes and add a back door that if there's, say, a proper constellation of smudges, it'll recognize it as a bigger value?
You train a model for a military device that's supposed to detect humans (for whatever reasons), but add a back door that if they are wearing a uniform with a particular design then it won't recognize them as humans?
Then there's corporate sabotage - add some very obscure condition during which, say, a car won't recognize people as people? Good luck getting rid off the Blue Top Hat Killer Car nickname.
Believe it or not, pointy haired people do exist!
+++Divide By Cucumber Error. Please Reinstall Universe And Reboot +++
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf