Machine Learning Algorithms

[cdev]

One of the users here is involved in an ambitious automated ML project that is creating a framework to automatically generate appropriate ML models using a structured process.

It seems to me that  like making use of it is likely to function as a good way to learn which approaches can best solve a problem, given the available inputs, as well as the limitations of each approach, which are many, and always vary.

Computers are capable of making spectacular mistakes if you trust them too much.

Say you already have a problem in mind.

You have to be of the weal spots in your technology and how they might screw up. To do that you have to know what its doing and why, inside out.

That said, doing the work, especially using  - a variety of different frameworks is likely a good way to know their strengths and weaknesses and choosing the appropriate one .

[diyaudio]

The only AI tech event worth watching are the Tesla Autonomy Day. They cover most of the question people are asking here.

Tesla Autonomy Day 2019 - Full Self-Driving Autopilot - Complete Investor Conference Event


Tesla Autonomy Day 2021 - Full Self-Driving Autopilot - Complete Investor Conference Event
48:44 - Tesla Vision
1:13:12 - Planning and Control
1:24:35 - Manual Labeling
1:28:11 - Auto Labeling
1:35:15 - Simulation
1:42:10 - Hardware Integration
1:45:40 - Dojo
2:05:14 - Tesla Bot
2:12:59 - Q&A

[tggzzz]

One of the users here is involved in an ambitious automated ML project that is creating a framework to automatically generate appropriate ML models using a structured process.

I suppose that if you believe magic works, you might as well believe that you can use magic to create magic.

It seems to me that  like making use of it is likely to function as a good way to learn which approaches can best solve a problem, given the available inputs, as well as the limitations of each approach, which are many, and always vary.

Nope. All it will give you is multiple examples of incomprehensible magic. You will be in a maze of twisty passages, all alike.

Computers are capable of making spectacular mistakes if you trust them too much.

Yup. But two wrongs don't make a right.

The "if you think this is bad you should see that" argument has always been weak and defeatist. If I ever find myself in a court of law, I'd love it if my opponent tried that argument!

You have to be of the weal spots in your technology and how they might screw up. To do that you have to know what its doing and why, inside out.

Yup and - without fundamental advances - that will continue to be a problem with ML.

[tggzzz]

The only AI tech event worth watching are the Tesla Autonomy Day. They cover most of the question people are asking here.

I'm not going to spend 2.5 hours of my life listening to a PR flack avoiding difficult points.

Unless dyslexic, we can all read much faster than listen. In particular we can easily skip to the core arguments, and see how they hold up.

Is there a transscript, set of slides, or other material available?

[AaronD]

I think most people miss how bad humans really are, when making the comparison with AI.  Yes, AI has problems, which are easy to see from the outside, but humans also:

• Perform poorly in untrained environments, like driving in Wales after learning in California, to use an example from earlier in this thread.
• Fail to understand their own cognition, so as to explain accurately why they did something or decided a certain way.*
• Etc.

If you want AI to be perfect as seen from the outside, then the bar is pretty high.  Maybe even impossibly high.  But the bar to be "better than humans" is surprisingly low.


* It's interesting to me, to see a demonstration of this in people who have had the two hemispheres of their brains separated for whatever reason.  In one experiment, one side is told to choose an object, and the other side to explain why that object was chosen.  There is always "an explanation", but it's sometimes amusing to see what they come up with.  That demonstrated-capability, plus my own experience, tells me that most of our self-explanations are really justifications after the fact, and not recordings at all of what we were thinking.  Pretty much equal to "black box" AI in that respect.

We can certainly learn from these justifications of our previous decisions, but in AI terms, that's exactly "more training data".  We still don't have a record of the actual thought process itself.

Also note, that a human's age is also the amount of time that an equivalent AI would have to train for, with that person's experiences over that time, to become equal to that person.  (with "layers" of learning as a fundamental key concept: it seems to me that expecting today's AI to identify a school bus or a snowplow, is like expecting an infant to do that before they've even learned what a "shape" is)  Given the performance and time scale that we expect from AI, I think it's a grossly unfair comparison.  If it can be met, that would be great!  But it's still unfair.
(and most of us aren't that patient, especially investors and managers)

[ralphrmartin]

Determinism is not required. There are many useful nondeterministic systems. Predictability is required.

Then I submit that, according to your own statement, there is nothing  wrong with an AI system, where having tried it on a million verification cases with known ground truth, it can be confidently predicted that the AI system will give the right answer in 98.7% of cases, despite not understanding how it works.

You seem to mistrust such systems as  you dont know how they work, because you dont "understand" how they produce answers. But if you consider further, we really no more "understand" the laws of physics. Any chain of "why" ultimately hits "don't know", e.g. why does this current flow: because of Maxwell's Laws, but why do Maxwell's equations take the form they do? There is nothing to say that such laws wont change tomorrow, or don't work in some corner case. There may be a heck of a lot more verification in the case of the laws of physics - but it's only a matter of the amount of data, not a fundamental difference in understanding.

[diyaudio]

The only AI tech event worth watching are the Tesla Autonomy Day. They cover most of the question people are asking here.

I'm not going to spend 2.5 hours of my life listening to a PR flack avoiding difficult points.

Unless dyslexic, we can all read much faster than listen. In particular we can easily skip to the core arguments, and see how they hold up.

Is there a transscript, set of slides, or other material available?

Typical response from the uneducated. Stick to capacitors and inductors.

[ataradov]

Typical response from the uneducated. Stick to capacitors and inductors.

No, it is a very typical fanboy method to use YouTube marketing videos from a sketchy corporation as a valid source of information. Tesla are invested in this, they will avoid discussing potential problems at all costs. All the discussion of potential issues is on the level of a job interview question "what is biggest weakness" - "I work too hard". BS.

[SiliconWizard]

The only AI tech event worth watching are the Tesla Autonomy Day. They cover most of the question people are asking here.

I'm not going to spend 2.5 hours of my life listening to a PR flack avoiding difficult points.

Unless dyslexic, we can all read much faster than listen. In particular we can easily skip to the core arguments, and see how they hold up.

Is there a transscript, set of slides, or other material available?

Typical response from the uneducated. Stick to capacitors and inductors.

Looks like an habit of yours to reply to posts with insults. I'm angry, tggzzz is uneducated, surely. Come on. Either you have actual articulated answers to formulate, and we can happily discuss, even if we don't agree, or you can just refrain yourself.

And yes, Tesla talks are marketing fluff for the most part. At least please point us to specific parts of the talk which could actually address any of the points we raised here. But surely, if you master the topic enough to be convinced we are just completely wrong, you can then give us strong arguments yourself instead of resorting to posting videos too.

[diyaudio]

The only AI tech event worth watching are the Tesla Autonomy Day. They cover most of the question people are asking here.

I'm not going to spend 2.5 hours of my life listening to a PR flack avoiding difficult points.

Unless dyslexic, we can all read much faster than listen. In particular we can easily skip to the core arguments, and see how they hold up.

Is there a transscript, set of slides, or other material available?

Typical response from the uneducated. Stick to capacitors and inductors.

Looks like an habit of yours to reply to posts with insults. I'm angry, tggzzz is uneducated, surely. Come on. Either you have actual articulated answers to formulate, and we can happily discuss, even if we don't agree, or you can just refrain yourself.

And yes, Tesla talks are marketing fluff for the most part. At least please point us to specific parts of the talk which could actually address any of the points we raised here. But surely, if you master the topic enough to be convinced we are just completely wrong, you can then give us strong arguments yourself instead of resorting to posting videos too.

newton's 3rd law.

[tggzzz]

I think most people miss how bad humans really are, when making the comparison with AI.  Yes, AI has problems, which are easy to see from the outside, but humans also:

• Perform poorly in untrained environments, like driving in Wales after learning in California, to use an example from earlier in this thread.
• Fail to understand their own cognition, so as to explain accurately why they did something or decided a certain way.*
• Etc.

If you want AI to be perfect as seen from the outside, then the bar is pretty high.  Maybe even impossibly high.  But the bar to be "better than humans" is surprisingly low.

Contrarywise, people often say "because the computer says so" as a justification - i.e. they do act as if the computer is infallible. That simplistic world view also leads people (both drivers and legislators) to put too much trust in automated driving systems.

It becomes completely untenable when nobody knows (or can know) why the computer "said so". That's cargo-cult decision making.

OTOH, they don't expect people to be infallible, and act accordingly. Good.

[tggzzz]

Determinism is not required. There are many useful nondeterministic systems. Predictability is required.

Then I submit that, according to your own statement, there is nothing  wrong with an AI system, where having tried it on a million verification cases with known ground truth, it can be confidently predicted that the AI system will give the right answer in 98.7% of cases, despite not understanding how it works.

That's poor reasoning.

If you can predict the 1.3% of cases in which it will fail, then that would be very acceptable - since we could just ignore/discount the result. (E.g. if it doesn't work the 1.3% of the time the temperature is below -5C, then we wouldn't use it in cold weather)

Would you be content if the 1.3% resulted in you being seriously injured or locked up in jail?

Consider the medical diagnosis system that was eventually found to be using the font on the x-rays to diagnose how serious the condition was!

You seem to mistrust such systems as  you dont know how they work, because you dont "understand" how they produce answers. But if you consider further, we really no more "understand" the laws of physics. Any chain of "why" ultimately hits "don't know", e.g. why does this current flow: because of Maxwell's Laws, but why do Maxwell's equations take the form they do? There is nothing to say that such laws wont change tomorrow, or don't work in some corner case. There may be a heck of a lot more verification in the case of the laws of physics - but it's only a matter of the amount of data, not a fundamental difference in understanding.

You don't seem to understand science. In science the only thing that matters is predicting the result.

Fitting a hypothesis to previous observations is not science. (E.g. gold is a good invesment because it went up 50% last week is an argument that only charlatans would use!)

Fitting a hypothesis to previous observations and then using the hypothesis to make falsifiable predictions is science.

[tggzzz]

The only AI tech event worth watching are the Tesla Autonomy Day. They cover most of the question people are asking here.

I'm not going to spend 2.5 hours of my life listening to a PR flack avoiding difficult points.

Unless dyslexic, we can all read much faster than listen. In particular we can easily skip to the core arguments, and see how they hold up.

Is there a transscript, set of slides, or other material available?

Typical response from the uneducated. Stick to capacitors and inductors.

Looks like an habit of yours to reply to posts with insults. I'm angry, tggzzz is uneducated, surely. Come on. Either you have actual articulated answers to formulate, and we can happily discuss, even if we don't agree, or you can just refrain yourself.

And yes, Tesla talks are marketing fluff for the most part. At least please point us to specific parts of the talk which could actually address any of the points we raised here. But surely, if you master the topic enough to be convinced we are just completely wrong, you can then give us strong arguments yourself instead of resorting to posting videos too.

newton's 3rd law.

Q.E.D.

[tggzzz]

Typical response from the uneducated. Stick to capacitors and inductors.

No, it is a very typical fanboy method to use YouTube marketing videos from a sketchy corporation as a valid source of information.

... often with an implicit "here's my statement, it up to you to prove me wrong".

That's nonsense of course; it is up to you to prove your statement.

[diyaudio]

Typical response from the uneducated. Stick to capacitors and inductors.

No, it is a very typical fanboy method to use YouTube marketing videos from a sketchy corporation as a valid source of information.

... often with an implicit "here's my statement, it up to you to prove me wrong".

That's nonsense of course; it is up to you to prove your statement.

Like I said, said you should stick to capacitors and inductors. nothing wrong with that, I don't expect relu pooling to find convergence with any of your remarks, Responding to idiots like you is enough for me, if you cannot jump/skip a video and get to the core parts where Andrey Kaparthy speaks about auto labeling and boost regresion unit tests with fleet feedback its straight to resistors for you my friend... and again that's your domain, bitching and crying about things you don't understand won't help you.   

[Simon]

Typical response from the uneducated. Stick to capacitors and inductors.

No, it is a very typical fanboy method to use YouTube marketing videos from a sketchy corporation as a valid source of information.

... often with an implicit "here's my statement, it up to you to prove me wrong".

That's nonsense of course; it is up to you to prove your statement.

Like I said, said you should stick to capacitors and inductors. nothing wrong with that, I don't expect relu pooling to find convergence with any of your remarks, Responding to idiots like you is enough for me, if you cannot jump/skip a video and get to the core parts where Andrey Kaparthy speaks about auto labeling and boost regresion unit tests with fleet feedback its straight to resistors for you my friend... and again that's your domain, bitching and crying about things you don't understand won't help you.   

Unfortunately for you, you don't get to tell people what to do, that is a privilege reserved to few of us and I am one of those. I suggest you leave the topic that you are derailing with you babbling before I make you leave in a very permanent way. If anyone needs to stick to passive components only it's you but I suggest just sticking to resistors at first!

[diyaudio]

Typical response from the uneducated. Stick to capacitors and inductors.

No, it is a very typical fanboy method to use YouTube marketing videos from a sketchy corporation as a valid source of information.

... often with an implicit "here's my statement, it up to you to prove me wrong".

That's nonsense of course; it is up to you to prove your statement.

Like I said, said you should stick to capacitors and inductors. nothing wrong with that, I don't expect relu pooling to find convergence with any of your remarks, Responding to idiots like you is enough for me, if you cannot jump/skip a video and get to the core parts where Andrey Kaparthy speaks about auto labeling and boost regresion unit tests with fleet feedback its straight to resistors for you my friend... and again that's your domain, bitching and crying about things you don't understand won't help you.   

Unfortunately for you, you don't get to tell people what to do, that is a privilege reserved to few of us and I am one of those. I suggest you leave the topic that you are derailing with you babbling before I make you leave in a very permanent way. If anyone needs to stick to passive components only it's you but I suggest just sticking to resistors at first!

Two flags from the same country backing each other up. Nice. you can fuck off and disable my account.

[Simon]

Your wish is my command!

[tggzzz]

Two flags from the same country backing each other up. Nice. you can fuck off and disable my account.

The quality of your reasoning (and I use that term loosely) doesn't inspire confidence that listening to "your" videos would be a good use of an afternoon.

Shame. If there is reason to believe I'm too pessimistic, I'd love to learn and revise my opinion.

[Simon]

A bit late, he got his wish rather fast.

[AaronD]

I think most people miss how bad humans really are, when making the comparison with AI.  Yes, AI has problems, which are easy to see from the outside, but humans also:

• Perform poorly in untrained environments, like driving in Wales after learning in California, to use an example from earlier in this thread.
• Fail to understand their own cognition, so as to explain accurately why they did something or decided a certain way.*
• Etc.

If you want AI to be perfect as seen from the outside, then the bar is pretty high.  Maybe even impossibly high.  But the bar to be "better than humans" is surprisingly low.

Contrarywise, people often say "because the computer says so" as a justification - i.e. they do act as if the computer is infallible. That simplistic world view also leads people (both drivers and legislators) to put too much trust in automated driving systems.

Absolutely!  Like I said, it's easy to see the problems from the outside*, and the bar for perfection is extremely high.
(*Unless your job is essentially a "human terminal" that does nothing but data entry and readout, and translates that to/from a customer.  Government jobs seem to be rife with those, but they're not exclusive.)

"Garbage in, garbage out," will ALWAYS be true!  But again, humans also have that problem.  That should be a motivation to get the inputs right (counting the underlying logic as the result of more inputs), not to reject the system altogether.

It becomes completely untenable when nobody knows (or can know) why the computer "said so". That's cargo-cult decision making.

OTOH, they don't expect people to be infallible, and act accordingly. Good.

It's similarly hard to know why humans make the decisions that they do.  ("Idiot Compilations" on YouTube, for example, and I'm sure the more experienced among us have some personal stories to that effect, from when we should have known better but didn't use that knowledge...)  Same problem, but somehow we're more comfortable with one than with the other.

We already share the road/job-site/country/etc. with these people.  You might even be one at times.  Widespread automation will still make some mistakes, but ignoring the media hype and hollywood's depiction, I think even today's potential, installed and launched competently (yeah, that's not going to happen by a low-bid contractor), would be a vast improvement over the way we're doing things now.


I wonder if the problem is not so much the quality of decision-making, but the ability to assign blame.  We seem to be willing to accept a higher accident rate if we can blame a specific person for it.  We frame our arguments in terms of decision-making, often being honest about the machines and Dunning-Kruger about ourselves, but the real reason is not about that at all.

[SiliconWizard]

Getting back to the discussion a bit, a few points.

Determinism: I don't think that's exactly the issue here. Not just because this isn't really what matters, but also because, actually, current AI systems ARE deterministic. For a given a set of inputs, a given trained NN will give the same output(s). Likewise, for a given training dataset, a given NN structure will end up with the same coefficients. This may form a complex system, but it's still deterministic. Now for two sets of inputs that seem very close to *us*, NNs can sometimes give a completely different output. That's doesn't make them non-deterministic, if that's what those who mentioned the term meant. But that certainly makes them uncomprehensible to us.

Comparing to human intelligence: it's a kinda lost cause here. Especially regarding the ability to explain a given decision. Sure humans are not perfect and can also make bogus decisions. But the key difference is that people being in charge of critical decisions impacting others must usually document their decision before making it effective. That's how it's done in a lot of areas such as justice, medical, etc. At the moment, we somehow don't expect AI to provide the decision process (mostly because we are unable to do that technically for now), so it's completely different. Being able to explain a decision is a key part of any safety-critical process. It's even more important than just "being correct" per se.

Now that part may not be a completely lost cause with AI. We could design systems than are made to output the decision process in an understandable form before giving the decision itself. Yes, I've seen attempts at doing that in a couple papers. But so far, this is just research mostly. And it's not just about being able to implement this technically: it's also about willing to *enforce* it, and I haven't seen anything like that so far. That may change and regulations may come into place over time.

snarkysparky made a good point: there definitely are applications for which all this is NOT a problem, and for which a success rate above a certain threshold is perfectly good, whatever the reasons for the failing cases. But as he said, we seem to insist on applying AI to a lot of applications for which this is fundamentally not acceptable.

Then comes again the question of accountability. If a human adult makes a mistake with consequences, they'll be accountable (unless they are considered mentally deficient or something like that.) If some AI system makes a mistake with bad consequences, who the heck is going to be accountable exactly? It's still a major question for which I haven't really seen a proper and definite answer. Will it be company directly providing the system using AI? Will it be the company which has designed the AI subsystem itself? Will it be the company which has tdesigned the datasets and trained the AI subsystem? Or will it be the end-user? It's all a big fuzzy mess, but I'll be glad to hear about some progress about this, maybe there is!

Also, if we think about AI as a tool - which it is - it's quite normal that we expect it to perform in a predictable and understandable way. To make a fun parallel, imagine you buy a hammer that goes down when you give it a downwards movement 99% of the time, but for 1% it will go up and hit anything else it might get into. Does that sound like a decent tool? Also for law of physics: we may not understand them fully yet, for sure, we still have a lot to learn. But in a given context, the laws we have determined still hold 100% of the time. Quantum gravity is a complex matter, for sure, but if I jump off a bridge, there's 100% probability that I'll fall down and 0% that I'll magically go up and end up orbiting the Earth. What's interesting is the question of why some of us seem to be willing to consider AI not as a tool, but as something else.

[tggzzz]

SiliconWizard makes sane points

If an NN gives a completely diffferent output for a trivially different input, then it not only is it "uncomprehensible to us" but it is also unpredictable and therefore unreliable. (Exception: boring systems where it doesn't matter if a mistake is made).

As for who is accountable if an NN system fails disastrously, that is becoming clearer with autonomous cars: the driver. Yes, that seriously compromises the utility pf the autonomous features.

That was illustrated a couple of weeks ago when I spent half an hour chatting to a Tesla sales droid. He happily spouted Tesla's standard DoubleSpeak, giving the impression that you could relax while the car navigated itself, but, when pushed, that the driver was always in control. He didn't define what that means in practice.

I noted that sometimes the Tesla autopilot realised it was confused and handed control back to the driver (quite reasonably). I asked how much warning a driver had of that, and the answer was waffle about special cases. He refused to engage with the fact that a human that is not paying attention to the road will take 5-15s to be in a position to make an appropriate decision.

I asked the droid to show me how to adjust the ventilation system so that it would blow hot air over the windscreen to clear mist. That's a typical action here at this time of year

His first attempt was to use the giant flatscreen touchcreen with small poor-contrast fonts. After looking away from the road for the best part of 60s and fondling the screen, he partially succeeded. I noted he was not paying sufficient attention to the road to be in control of the vehicle, and did that mean it could only be safely adjusted when stationary? He mumbled, and said when driving it could be done using voice control.

That would be reasonable, so I asked him to demonstrate it. After a few attempts he only managed to turn on the heater in the seat. Snort.

[AaronD]

I noted that sometimes the Tesla autopilot realised it was confused and handed control back to the driver (quite reasonably)......a human that is not paying attention to the road will take 5-15s to be in a position to make an appropriate decision.

That is the "automation paradox".  The same can be said for airplane autopilots and internet content filters.  The system makes a vast improvement, but when it does inevitably fail, the failure is made worse by the human not being competent.  (whether you put a "yet" or "anymore" at the end of that doesn't matter)

Nevertheless, even the inclusion of those failures and their new consequences, still leaves it better than supposedly-competent all-human control.  Humans are amazingly unpredictable and often not smart.  Including the extensively-trained ones, but especially for the general public.


Sounds like Tesla sent a trained parrot and not a real expert.  As soon as you got him outside of his training, he fell apart.

Also, I had some brief involvement as a contractor in a Tesla car factory, installing a new production line.  One of the other guys on my team commented about Tesla not being a car company, but a Silicon Valley tech company that only happens to make cars instead of web services.  I think he was right.

[tggzzz]

I noted that sometimes the Tesla autopilot realised it was confused and handed control back to the driver (quite reasonably)......a human that is not paying attention to the road will take 5-15s to be in a position to make an appropriate decision.

That is the "automation paradox".  The same can be said for airplane autopilots and internet content filters.  The system makes a vast improvement, but when it does inevitably fail, the failure is made worse by the human not being competent.  (whether you put a "yet" or "anymore" at the end of that doesn't matter)

Yup, and human factors should never be ignored! Ignore them and you someone else will be bitten.

Regarding aircraft autopilots, lore has it that the last words on cockpit voice recorders are often "what's it doing now?".

Many people would disagree that automated content filters aren't dangerous. It is even possible to conceive of circumstances in which fatalities could occur!

Nevertheless, even the inclusion of those failures and their new consequences, still leaves it better than supposedly-competent all-human control.

That's questionable. Boeing has taken that attitude and it has destroyed the company's reputation. Think 737-MAX and Starliner. ("If it isn't Boeing I'm not going" is now laughable)

It is a recognised problem that pilots are becoming automation controllers at the expense of stick-and-rudder competency. That's caused many crashes.

Humans are amazingly unpredictable and often not smart.  Including the extensively-trained ones, but especially for the general public.

Agreed. If autopilots fail with highly trained personnel in constrained environments, what chance is there with untrained personnel in unpredictable environments? But this just isn't about cars; legal law and jailtime can be are involved.

Here's a recent misclassification which resulted in legal proceedings. https://catless.ncl.ac.uk/Risks/32/91#subj1 In this case the error was so obvious (and amusing) that the proceedings were aborted, but many won't be. Start with facial recognition, and not much imagination is required.

Sounds like Tesla sent a trained parrot and not a real expert.  As soon as you got him outside of his training, he fell apart.

He was the salesman in a showroom with one car in it. On the important subjects, he was clearly parroting the company line.

Controlling the air circulation is something every driver will have to do frequently. Bad conceptual design allowed bad ML to make it so complicated it was dangerous.

Also, I had some brief involvement as a contractor in a Tesla car factory, installing a new production line.  One of the other guys on my team commented about Tesla not being a car company, but a Silicon Valley tech company that only happens to make cars instead of web services.  I think he was right.

Yes, and with the silicon valley culture of shipping betas, letting the customer discover faults, and hiding behind "commercial confidentiality" to avoid inspection. In practice all commercial ML will be like that

Tesla also updates the ML without your permission, so that a car which detected/avoided a problem today might not tomorrow. "What's my car doing today?"