| General > General Technical Chat |
| Planting Undetectable Backdoors in Machine Learning Models |
| (1/2) > >> |
| SiliconWizard:
Nice! :popcorn: https://arxiv.org/abs/2204.06974 |
| TomKatt:
These AI type applications have become so complex, with datasets so large, that human minds can no longer understand their operation. And if you don't understand how something works, how would you ever really be able to identify errors or malicious activity? --- Quote from: Big Think ---The computers that run those services have programmed themselves, and they have done it in ways we cannot understand. Even the engineers who build these apps cannot fully explain their behavior. --- End quote --- https://bigthink.com/the-future/black-box-ai/ |
| tszaboo:
So what is exactly the danger with this? As I understand this it would work the following: Having two models, one doing the expected operations, one the backdoor. So for example if you pass an image of a "blue stopsign next to a road" to an image classifier, object detector, it will tell you it's a cat. How can they exploit this? |
| tom66:
--- Quote from: TomKatt on March 01, 2023, 12:51:40 pm ---These AI type applications have become so complex, with datasets so large, that human minds can no longer understand their operation. And if you don't understand how something works, how would you ever really be able to identify errors or malicious activity? --- End quote --- This is the same issue with autonomous vehicles. The prediction by some was that we could build a neural network that takes an image in, runs it through the network, and generates steering, accelerator, brake, etc instructions. Unfortunately, that's effectively impossible to prove safe. The reality is that every successful autonomous vehicle out there uses the NN to process images, and solve the path and free space problem, but ultimately that data is processed by an ordinary algorithm running on an ordinary CPU, with that algorithm having been created by engineers sitting at their desks. |
| AndyC_772:
The example they give is about a hypothetical NN used to assess creditworthiness and approve or deny bank loans. The 'clean' NN would make a reasonable attempt to decide whether or not someone should be offered credit, based on the information available to the bank at the time. The 'backdoored' NN would, however, be made to unconditionally approve a loan, if some very particular, carefully crafted conditions were met. For example, a loan of $10,000 might be rejected, but a loan to the same individual of precisely $10,004.26 would be approved. Crucially, the agency that performed the training would be able to know what criteria would result in these 'false' approvals, but it would be mathematically very difficult to detect that they existed without that inside knowledge. |
| Navigation |
| Message Index |
| Next page |