EEVblog Electronics Community Forum
Products => Computers => Programming => Topic started by: Picuino on February 11, 2022, 12:04:40 pm
-
New AI capable of entering programming contests and beating 50% of humans (for now).
https://www.deepmind.com/blog/article/Competitive-programming-with-AlphaCode (https://www.deepmind.com/blog/article/Competitive-programming-with-AlphaCode)
https://www.youtube.com/watch?v=IVBKRJq1bxI (https://www.youtube.com/watch?v=IVBKRJq1bxI)
How Programmers have become Obsolete - Deepmind AlphaCode
https://www.youtube.com/watch?v=t3Yh56efKGI (https://www.youtube.com/watch?v=t3Yh56efKGI)
AlphaCode Explained: AI Code Generation
-
I call BS. They use some sketchy metrics for performance. There is no point in comparing the code to "average" person on some random platform. Plus they always make it code trivial, self contained and purely algorithmic things. And then somehow extrapolate it to replacing jobs.
This is like Tokamak of programming. They would work on it for 70 years with no real practical result, and the breakthrough is always going to be 5-10 years away to secure funding.
-
This *is* pure BS. =)
-
So none of you tried GitHub CoPilot I see?
AI is already here, I use it (technical preview) and am better for it.
Saying competitive programming is just trivial and self contained makes me laugh. You've never tried, have you?
Try out this year's Advent of Code, check the time for the first 1000 entries. Can you do better than 50% of them? (I am generous with top 1000, as probably not all of them would actually enter a competition.)
Why can't we just take joy in this awesome field of applied ML/AI?
Ok, the clickbait title is cringe, but the actual news is nothing short of awesome!
-
So none of you tried GitHub CoPilot I see?
AI is already here, I use it (technical preview) and am better for it.
I've seen other people try it and I'm, not impressed at all.
Saying competitive programming is just trivial and self contained makes me laugh. You've never tried, have you?
It has nothing to do with real jobs.
Why can't we just take joy in this awesome field of applied ML/AI?
When it turns into something practical, for sure. Right now it is no better than crypto/NFT stupidity. A lot of hype, no actual applications other than scamming investors for money.
-
Just for the sake of discourse, I agree with all of your statements :) Though, CoPilot is much better when trying it yourself, you should try it when you get a chance! With Rust and golang it is awesome! I am underwhelmed with C++ still though.
-
I'm not interested in trying it myself at all. It is impressive in the same way as watching crows figure out logic puzzles is impressive. Kind of cool, but completely impractical for anything real. I may be proven wrong, of course, but I seriously doubt it, so I'm not wasting my time on that.
Also, CoPilot only works because it had access to a huge code base during training. And its continuous improvement relies on having access to even more code. And if a lot of the new code is generated by the AI, the quality of the future code will start degrading.
And I think it relies on user feedback on the quality of the generated code for improvements. And that's dangerous - relying on the feedback of random people. I see racist Microsoft bot situation repeating.
-
Try out this year's Advent of Code, check the time for the first 1000 entries. Can you do better than 50% of them? (I am generous with top 1000, as probably not all of them would actually enter a competition.)
I do advent of code. It's great. I always learn something useful and it makes me a better programmer. But it isn't actually representative of day to day professional software development and none of the people who do well in it would argue otherwise. In real life I'm never limited by the ability to type some barely working code in 7 minutes that only has to get the right answer for a single input. And while a human can definitely take some interesting trick they learn doing competitive programming and apply it to more complicated projects, AI bots aren't humans and they almost never can generalize behavior in that way. That's basically like saying that because computers can multiply numbers several billion times faster than humans that they are better at proving theorems. In reality the programs used for theorem proving are orders of magnitude more complicated than multiplication and only work when carefully set up for a specific problem class by a human
I haven't used github copilot. I have seen the demos and know some people who have at least tried it, although none who use it regularly. As a fancy form of autocomplete maybe it or something like it will be useful to some people, but again, I'm just not limited by how fast I can type, and that is really all it's offering.
-
Saying competitive programming is just trivial and self contained makes me laugh. You've never tried, have you?
Try out this year's Advent of Code, check the time for the first 1000 entries. Can you do better than 50% of them? (I am generous with top 1000, as probably not all of them would actually enter a competition.)
I haven't done Advent of Code or TopCoder or the ACM student contest.
What I *have* done:
1) ICFP Programming Contest.
Annual 72 hour online contest, usually to solve a single task. Often there is a "Lightning" Prize for the best solution (to the full problem) submitted in the first 24 hours. I was captain of teams (sometimes just me and one other, sometimes up to three others) called usually "Dylan Hackers". I/we won the following prizes:
2001: 2nd place -- rewrite documents in an HTML-like markup language to minimise their size i.e. find the shortest markup that gives each element the same properties as the original. Programs were submitted in executable form and the judges ran them on previously-unseen inputs. Execution time is a tie-breaker in the event of same output size. Any crash or failure to finish within a time limit disqualifies. (Always be ready to output the current best version, even if it's unchanged from the original!)
2003: Judge's prize -- given a known set of race tracks and a car with certain acceleration, braking, and cornering physics properties, submit the driver control inputs that minimise the 1st lap time from a standing start. The judges did *not* run your program. We used a hybrid approach, writing a simulator which I then drove manually, and an optimiser to incrementally improve the solution. The winner used several hundred Pentium 4 PCs at a university to brute-force a slightly restricted solution space. Our solutions were significantly better on about a 1/3 of the tracks, very very similar on almost all the rest, but we were absolutely slaughtered on one (not a normal race track, but a half-toned photo of Phil Wadler's face)
2005: 2nd place AND also Judge's prize -- given a map of part of Chicago with a number of banks, there is a robber trying to visit banks and steal money from them and a set of police trying to catch the robber. Write AI for the robber, and AI for the police, and compete in a double round-robin against other entrants, sometimes being the robber and sometimes the cops. The 2nd place was for the initial competition, and then a few weeks later they had a surprise 24 hour competition based on the main one but with significantly changed specs. The Judge's prize was given as the entry that was easiest to update for the changed specs.
Samsung internal programmer testing. You need basically a Masters'-level CS education to get hired in the first place, but then they have internal competitions in a series of 3-hour exams at four graded levels. Each exam consists of solving a single problem in C or Java (restricted to basic libraries e.g. in C stdio and stdlib). As I reverse-engineered by looking at historical tests the stages were roughly 1) you can use a direct, greedy, algorithm; 2) there is a search space but it's small-enough to brute-force it; 3) like for the second stage but the search space is too large without pruning it, "dynamic programming" is needed; 4) wild-card ... just super-challenging. I passed at the 4th level. In the Moscow office about 1% of programmers did that. World-wide the percentage was quite a bit lower.
I'll describe the level 4 exam task I passed. You have to write two functions. The first function takes a grayscale bitmap image (I think maybe it was 1024x1024, 4 bpp) and a larger bitmap (4096x4096?) that you should somehow recode the grayscale image into. The driver program then takes your bitmap image, applies a random series of 90ยบ rotations and mirror image operations around the horizontal and vertical and two diagonal axes. And then it randomly corrupts up to N bit values in each 4x4 cell. And then calls your second function. The job of your 2nd function is to recover the original image from the corrupted bitmap.
Here is a level 3 one. There is a set of some thousands of nodes and arcs in a fully-connected graph with no cycles. Each arc is a resistor, each with different values. You may take any one resistor, disconnect it, and reconnect it between any two new points. Which choice minimises the maximum resistance between two points in the graph. (Remarkably, you can do this one in O(N) time and space)
So ... yeah ... I'd like to see an AI tackle any of these, purely from the English text description of the problem.
Other than by saying "Oh, this problem was already given in X competition in Y year, and here's the winning entry..."
-
I haven't used github copilot. I have seen the demos and know some people who have at least tried it, although none who use it regularly. As a fancy form of autocomplete maybe it or something like it will be useful to some people, but again, I'm just not limited by how fast I can type, and that is really all it's offering.
I know I say this on a regular basis but... yes, I don't get it either: why so many people (but are there really that many, or do the numbers look just inflated due to these persons being more "loud"?) seem to be so stuck on the "typing" aspect of programming, and go at sometimes great lengths to reduce the typing (just look at all those languages trying to limit typing using cryptic keywords, symbols and constructs, for instance), while this is actually a minor fraction of your work. Actually, if the typing part of your work is significant, I don't even want to look at your code, and probably even less so use it.
As to these automated tools in general, I find them more annoying and distracting than useful. Even just the auto-completion in many recent editors - that's just a pile of crap. That annoys me to no end. I don't want some f*cking thing popping up all the time telling me what I'm supposed to be doing or typing. Completion itself can be useful, but I want to trigger it myself if needed, not pop it up by itself at almost every key press. It sure is down to personal preferences I suppose - I for one find it *extremely* annoying.
"Copilot" is a well devised term actually: I personally don't want to have something that will keep telling me what I'm supposed to do or want. And if you don't know what you're doing to the point of needing that, you probably should pick another job. Just my 2 cents. =)
-
Is it really "us" that are loud? At the time of my writing in favour of AI (and in this instance AlphaGo in particular) there were two pretty loud comments before me.
It's not about speed. It's about confidence.
If I implement behaviour a in a way that matches what most other quality projects does, then I can feel confident it's at least not stupid. I still have full autonomy to insist that my version is the way to go. Maybe I have domain knowledge etc. Actually, if the not-using-tools part of your work is significant, I don't even want to look at your code, and probably even less so use it.
There's nothing annoying with AlphaCode, you configure it as you like. I for one, use vim (not at all about typing speed, but editing speed) and I bind the keys I want to do what I want when I want it to happen. Crazy idea, right !? :P
True code completion (that is context and intention aware) is actually a speed boost too though, I am a quick typer but an easily distracted thinker/multitasker. If things can happen almost as fast as I think them, then I win a lot because I don't get lost in the "where was I" phase if the train of thought speeds away from me.
I admit going fast is fun (and helpful for me). I do a lot of in-house demoing and tutoring, and tools like these help me make live coding ... not boring.
View CoPilot as a pair programmer with deep knowledge of the solution space and conventions, but little knowledge of the exact domain.
When tests pass, linters(/static analyzers) are silent, CI/CD pipelines are green and the AI is happy - then I am happy. They all bring something to the table. They are tools. They are here to stay.
The annoying aspects laid out above, simply tells me they are speculation and not based on experience.
I did not have time to make this shorter, sorry.
-
As to these automated tools in general, I find them more annoying and distracting than useful. Even just the auto-completion in many recent editors - that's just a pile of crap. That annoys me to no end. I don't want some f*cking thing popping up all the time telling me what I'm supposed to be doing or typing. Completion itself can be useful, but I want to trigger it myself if needed, not pop it up by itself at almost every key press. It sure is down to personal preferences I suppose - I for one find it *extremely* annoying.
I absolutely hate them. Maybe they're useful for people who are working on code bases with huge APIs that they barely know at all -- I imagine in the Java business app world, or websites using three thousand JavaScript libraries imported from God knows where (and used live from the original place, so when some guy deletes his trivial five line one function library to trim trailing spaces from a string, half the internet falls over -- it happened...)
Emacs has a built-in library function dabbrev.el, normally triggered by "alt /" (M-/ in emacs-speak), which attempts to complete the current word by looking at your existing code. First it searched backwards in the same file for the first matching word. If none is found it searches forward. If still none is found then it searches in other files you have open. If you don't like the completion it found then you can immediately hit M-/ again and it finds the next match.
I find this is a pretty good language and context-agnostic level of assistance. Sure, it doesn't help much with looking up APIs (though it will find things in header files you have open), but it's very useful for avoiding repeatedly typing long descriptive variable and function and field names.
And it works just as well when you're writing documentation -- or a novel -- as it does writing program code.
[you can also hook emacs up to compiler-assisted API lookup for various languages if you *like* the modern IDE style, but even then you have full control over how and when it is triggered]
-
Completion itself can be useful, but I want to trigger it myself if needed, not pop it up by itself at almost every key press.
I find it useful in the same was as a real-time preview window: if I'm not getting the completion hints I've probably mistyped or got the wrong stuct or similar things. Unless it's a looong name or something I'll just carry on typing it myself, but just like code, finding and fixing typos early is cheaper in the end :)