AI destroying economic system

[PlainName]

Indeed, but isn't that just statistics? It detects more than us because it can go through more than us and be tireless about it. But does it understand things any more than just knowing the likelihood of C following B?

You’re closing yourself the path to answering the question by describing these things as binary options that exclude each other. They do not. There is even little reason to think that humans are anything but sophisticated statistical machinery.

Well, maybe. I don't know how to ask the question in a better way. I think that humans are a bit more than sophisticated stats machine since (most!) can figure something doesn't look right and use a different approach to verify things. Although, as before, that's a simplistic example of what I'm trying to put across.

The important distinction is that there are different kinds of statistical machinery. They’re not better or worse, just different. [...]

Useful and informative, thanks.

[Randy222]

Well, maybe. I don't know how to ask the question in a better way. I think that humans are a bit more than sophisticated stats machine since (most!) can figure something doesn't look right and use a different approach to verify things. Although, as before, that's a simplistic example of what I'm trying to put across.

A year or two ago I saw some news bit on some medical site about the "AI Robot" doing some types of surgery, in the writeup it said the bot is so smart that it could detect a mistake it made and correct it !!!

Just as humans made many many engineering mistakes over the past 1,000's of years, even AI will make mistakes. Whether such mistakes are seen as "doesn't look right" or not, will not negate the error. It's either corrected early on or the mistake goes on for days weeks years and then something bad happens (bridge falls, buildings collapse, planes crash, code is compromised, etc etc).

[golden_labels]

Well, maybe. I don't know how to ask the question in a better way. I think that humans are a bit more than sophisticated stats machine since (most!) can figure something doesn't look right and use a different approach to verify things. Although, as before, that's a simplistic example of what I'm trying to put across.

There is no need for you to excuse yourself. That was not a criticism. I was pointing to a mistake I blieve you made, so you could overcome it.

[booscrawl]

If you think about it very simply, as a data preservation engine, then LLMs are perhaps the best way we could possibly store an enormous archive of tens of millions of ancient books. Many, many terabytes of scanned pages, compressed down to a few hundred gigabytes of weights, capable of regurgitating any of the training corpus at any time when merely prompted with the first page or title of the book.

Ok, I'm done typing, and I have not been able to contain my laughter at what I just wrote. It's not a very good solution for archival. It is a lot better than letting books dry-rot.

[golden_labels]

Except that LLMs don’t store original data. They only hold statistical picture of the entire learning set. Cutting off a large part of the original information is the very goal of neural network algorithms. This is how they can store so much information in so little space, and the source of their ability to react to new inputs.

So the content of these books is not preserved. Unless we consider this picture in attachment to be a preservation of your words.

[Randy222]

If you think about it very simply, as a data preservation engine, then LLMs are perhaps the best way we could possibly store an enormous archive of tens of millions of ancient books. Many, many terabytes of scanned pages, compressed down to a few hundred gigabytes of weights, capable of regurgitating any of the training corpus at any time when merely prompted with the first page or title of the book.

Ok, I'm done typing, and I have not been able to contain my laughter at what I just wrote. It's not a very good solution for archival. It is a lot better than letting books dry-rot.

As mentioned, preservation in digital realm needs 1) a copy, 2) a signed something to verify the copy was not changed.
The LLM's dont do any of that.

In the current ecosystem, LLM's are often biased and carry some error. But in bigger picture you can reduce the notion of bias and error by doing more ML on the "same" dataset where such dataset has been altered.

In digital world, deterministic "right" & "wrong" becomes exponentially harder to decipher.

[tom66]

If you think about it very simply, as a data preservation engine, then LLMs are perhaps the best way we could possibly store an enormous archive of tens of millions of ancient books. Many, many terabytes of scanned pages, compressed down to a few hundred gigabytes of weights, capable of regurgitating any of the training corpus at any time when merely prompted with the first page or title of the book.

Ok, I'm done typing, and I have not been able to contain my laughter at what I just wrote. It's not a very good solution for archival. It is a lot better than letting books dry-rot.

There's an interesting link between compression and intelligence.   The "best" LLMs are those models which exhibit the highest apparent "intelligence" for the lowest size of the model.  You could take petabytes of training data and perhaps produce a 1-2T class model that is comparable to Claude Sonnet.  Or then further reduce it to something like Claude Haiku (likely 100B-class).  Such a model would be able to recall vast amounts of knowledge, and could be considered an expert at generalised knowledge.  However, even such an encoding could not include all knowledge, and the model needs to be able to access the internet for obscure knowledge that did not make its way into the model.

Good 3b1b video on the link between compression and intelligence.  Also, I'm certain that Claude is so-named because of Claude Shannon, the father of information theory, and the link to compression.

[golden_labels]

As mentioned, preservation in digital realm needs 1) a copy, 2) a signed something to verify the copy was not changed.
The LLM's dont do any of that.

In the current ecosystem, LLM's are often biased and carry some error. But in bigger picture you can reduce the notion of bias and error by doing more ML on the "same" dataset where such dataset has been altered.

In digital world, deterministic "right" & "wrong" becomes exponentially harder to decipher.

If this “mentioned” refers to my post, then I didn’t consider the second point (digital signature). I don’t find them useful for general preservation either, though I’ll skip that tangent now.

As for the remaining part, it’s not a matter of the current ecosystem. As in: it’s not a temporary situation, some kind of shortcoming, a thing to be repaired. It’s the opposite way. It has always been the ultimate goal to maximize the amount of data removed. This is the very basis of operation, not an unwanted artifact. I said “always” and I emphasize this: I didn’t use such a heavy quantity frivolously.

Neural networks, LLMs included, are statistical models. In model calculation, the datapoints aren’t the data modelled. They are a sample of the data modelled. The actual, true distribution remains obscured. Our model is meant to approximate that unknown distribution, not the sample. If a model matches the sample too well, we have overfitting and that makes the model garbage. If LLMs stored the training data, they’d perhaps recall it perfectly, but the very purpose of building them would be missed. We’d

The corollary to above is something that may seem counterintuitive at first. Hallucinations are a feature, not a bug.

[paulca]

There's an interesting link between compression and intelligence.   

In human and animal terms, the bandwidth of the senses is orders of magnitude higher than the brain can properly process, so everything is compressed, lossy compression.

The "more intelligent" an animal can be partly down to how well it compresses reality into what it can process.  Get that wrong and you start with poorly selected information. You make poor decisions, you have poor outcomes, you are not biologically successful.

In the cereberal cortex terms it's much the same thing.  Which parts you select out of what your sensory cortexts gives you and encoding that into memory ... the better you are at that, the more effiecient you use information, the more "intelligent"?

[Siwastaja]

Neural networks, LLMs included, are statistical models.
...
The corollary to above is something that may seem counterintuitive at first. Hallucinations are a feature, not a bug.

Just like human brain, so there's nothing counterintuitive in it. LLM hallucinations are very similar to human hallucinations. AI slop is very similar to human slop. Also, AI being thorough and careful is very similar to human being thorough and careful.

I have caught myself behaving like the early / crappy LLMs a few times lately. This is because with all the AI discussion, one starts to notice things. For example, I have written deg C, when I meant deg (as in, an angle, not a temperature); when speaking, my brain similarly autofills, sometimes incorrectly.

This is human brain being a statistical machine, which chooses a token, often a part of a word, which is most probable continuation to the previous tokens. It's human brain generating speech/writing "on autopilot" - low thinking effort.

Human created AIs (LLMs) as their own image.

We are stochastic parrots, and I'm not sure if our intelligence is "true intelligence", given that it's now trendy to question the trueness of artificial intelligence.

The scary thing is, human brain is not evolving (at any relevant speed in our lifetimes), but AI is. Just a year ago agentic AI doing science and engineering required a lot of babysitting and hand-holding. Hallucinations and slop were a thing. Now, suddenly the quality is so high that we just laugh at the human slop and human hallucinations, and wonder why we would use humans for any programming, software design, engineering work at all, besides some sort of very high-level intuition, but even that is becoming questionable as AI is now capable of designing whole complex systems on its own.

But, AI-generated art I don't like, and also, I don't like reading AI-written articles, something bugs me in the output style. And it's not only model-specific mannerisms, all LLMs seem to share something common, some sort of internal thought pattern leaks out, and is in the uncanny valley. It walks and quacks like a human, but isn't. But, when it tells me it finished a job which I always wanted done but which previously would have cost $100000 and 6 months, in 30 minutes and for $50, AI mannerisms do not bother me at all, I grin from joy every time.

Hallucinations are now a feature of humans, almost exclusively; human taught AI to fact-check itself, and tools and resources to do that, but we poor humans are still under-resourced and slow enough that we sometimes just create some slop because we can't afford checking. At least I'm guilty of that; my shitposting here is most often without tools, from my neurons alone, and as a result, I'm sometimes wrong.

AI makes fewer and fewer mistakes every month, and that's nearly unbelievable, but it's true; anyone can observe it themselves.

[Randy222]

Good 3b1b video on the link between compression and intelligence.  Also, I'm certain that Claude is so-named because of Claude Shannon, the father of information theory, and the link to compression.

IMHO, Andrey Markov is of greater significance when it comes to digital input/output. But, my opinion is shadowed by others who do believe Shannon is. So be it. I like the quick convergence to the probable outcome using as little input data as possible.

[SpacedCowboy]

AI makes fewer and fewer mistakes every month, and that's nearly unbelievable, but it's true; anyone can observe it themselves.

Yesterday. I want to repeat that, Yesterday I got Claude to help me add matrix operation auto-detection to my compiler. The Apple M4 (and later) range have the SME extension to their cpus, which is a high-speed matrix multiplier. That means the compiler could auto-detect matrix operations (basically 2 nested loops calculating an array based on 2 other arrays) and schedule it on the SME co-processor.

The time taken in c++ (clang -O3) to run 10,000 128x128 matrix multiplies on the M4 Max was 10.94 secs, reasonably respectable. That dropped to 0.07 secs with the SME - giving ~1.3 TFlops from normal floating-point array multiplication, no special source-code or libraries.

It took 2 (count them, 1,2) days to build something I have never seen before in a programming language but always wanted, clean interoperation of cpu and gpu, with the compiler managing any data-hazards. If anyone has ever tried compute shaders before, you’ll know all the boiler-plate you have to go through to get the two devices talking, and the scheduling and memory-barriers etc to get data safely where you next need it.

Here’s a parallel Mandelbrot set calculation that can execute on one of

• The cpu
• The gpu
• SIMD cpu

- you might have to look hard to find what’s different from any normal Mandelbrot algorithm, but there’s 1 line which tells the compiler that this is a data-parallel block of code, and it’s:

Code: [Select]

par mandel :reduce(+ total) {…}
‘Par’ introduces a data-parallel block, ‘mandel’ is just a name, and if you want any reductions (all computations updating a single variable) you declare them afterwards. That’s it.

The compiler will then create multiple versions of the code block, one for each of the above list. At runtime you can say “run block mandel using this method”, again, any of the above methods, with the default being ‘auto’ - the fastest. With auto set, it’ll measure one pass through the loop for each method, then run your complete computation using that. For gpus it creates MSL (Apple), PTX (Nvidia) and SPIR-V (AMD, Android) forms of the code block you specified as ‘par’, and auto-detects which hardware it should run, again, at runtime.

Honestly, even if I say it myself this is a game-changer for fast data-parallel code. You use the same language, there’s no marshalling of data, and the results speak for themselves.

So yes, I agree, these things are getting better and better. They still need to be driven with ideas, they’re really good at creating connections, pulling in knowledge, and executing on test-driven results, but they’re not (yet ?) creative enough to say “hey, why don’t we try and attack this problem by doing…”, at least in an engineering sense. They are amazing at the other bits though, and I wouldn’t have expected to be able to do the above 6 months ago.

[NorthGuy]

Honestly, even if I say it myself this is a game-changer for fast data-parallel code. You use the same language, there’s no marshalling of data, and the results speak for themselves.

Interesting. Soon there will be no languages, as AI will be able to write assembler by themselves, and then soon afterwards they will just generate machine codes directly. So, your language may be the last language ever developed ...

[Randy222]

Honestly, even if I say it myself this is a game-changer for fast data-parallel code. You use the same language, there’s no marshalling of data, and the results speak for themselves.

Interesting. Soon there will be no languages, as AI will be able to write assembler by themselves, and then soon afterwards they will just generate machine codes directly. So, your language may be the last language ever developed ...

Doesnt there need to be an instruction to AI before it could generate the lower level code?

[NorthGuy]

Doesnt there need to be an instruction to AI before it could generate the lower level code?

English. Someone gives instructions. AI generates CPU instructions. Everything that was in between was designed for humans so they could write programs, but it is redundant for AI.

[Siwastaja]

Doesnt there need to be an instruction to AI before it could generate the lower level code?

English. Someone gives instructions. AI generates CPU instructions. Everything that was in between was designed for humans so they could write programs, but it is redundant for AI.

Yes and no. AI is still human-like in how it thinks. It can definitely write direct machine code out of English instructions, already today, but just like it does for human, it requires more effort to do so. Layers that abstract repetitive mechanical tasks are equally beneficial to humans and AI. Generating direct machine code on generic large neural network (human brain or LLM) requires a lot of processing. Spitting machine code out of intermediate higher-level code, which can be parsed by a simpler and way more efficient thing called compiler made sense in the past, and probably keeps making sense in the future.

The pattern is visible today if you ask AI to do some task which requires repetitive, boring, resource-consuming actions: it writes a small python program to perform those! Even Python is way more efficient than LLM doing it all. So it avoids using the expensive electronic brain, and cluttering expensive context with that detailed work.

AI is goal-oriented though, and largely skips social constructs, so it may choose differently what programming languages, what kind and how many layers it will use, and to the horror of many professional software developers, it tends to choose a simple, self-written solution. Not in direct machine code, though.

I'm not completely ruling out the possibility it would start writing assembler or machine code directly in the future, but currently I think it will first try to use most suitable existing language + compiler, or secondarily, will write some simple language specification and compiler for that, instead.

[golden_labels]

Doesnt there need to be an instruction to AI before it could generate the lower level code?

Not if that was in the training set at a quantity and quality sufficient to build the relevant patterns. Word “language” in Large Language Model doesn’t stand for a human language or a natural language. It’s a more abstract concept which extends way beyond what most humans recognize as a language. The first subject I got for my degree thesis⁽¹⁾ back in 2000s was a language model for network traffic. Even the earliest GPTs released to the public were able to generate ASCII art, including correct TUI interfaces with meaningful and controllable information. That is also correctly modeled by as a language. So is computer code, regardless of the vocabulary used.

The question is not if it’s technically possible, but if anybody is wishing to pay for it. There is little reason to generate code directly. Doing the opposite, though…

⁽¹⁾ I rejected this idea and went another path.

[Randy222]

I have used plenty of AI to help write just code in basic languages, bash, python, etc, and in majority of the cases, the code generated was not optimized or simply did not work.

I use MS CODE app with some plugins and AI integrations, it does a fairly good job of highlighting my mistakes, which I can then OK it to fix for me, or I do it myself.

If I am relying on AI to write good-to-go code at any level, it better damn well be good-to-go, which from what I have seen, not there yet. In some environments, +1nsec matters.

[nctnico]

My results with the paid version of Claude code have been stellar. And the same goes for other software engineers I work with. You still need to sit in the driver's seat though. The key is to start out with a clear goal and create a plan. Then let that plan be reviewed by Claude's review agent. Once you are completely happy with the plan, only then let Claude generate code. Typically I start by writing a coherent project spec in a text editor and paste that into Claude's desktop tool.

[NorthGuy]

My results with the paid version of Claude code have been stellar. And the same goes for other software engineers I work with. You still need to sit in the driver's seat though. The key is to start out with a clear goal and create a plan. Then let that plan be reviewed by Claude's review agent. Once you are completely happy with the plan, only then let Claude generate code. Typically I start by writing a coherent project spec in a text editor and paste that into Claude's desktop tool.

I managed to pass a lot of tedious job to Claude 5.5 Sonnet and used very few tokens. I'm not sure, but I feel that even a free version would be enough. This is certainly very beneficial. Now I don't need to hire anyone to do this job for me. I am now planning to do more things that I otherwise wouldn't do at all. I'm also sure that AI affects my customers just as well. So, the net effect is very hard to predict.

[paulca]

My results with the paid version of Claude code have been stellar. And the same goes for other software engineers I work with. You still need to sit in the driver's seat though. The key is to start out with a clear goal and create a plan. Then let that plan be reviewed by Claude's review agent. Once you are completely happy with the plan, only then let Claude generate code. Typically I start by writing a coherent project spec in a text editor and paste that into Claude's desktop tool.

In work we use a couple of system on top of each other, if you like that approch.

Analyse -> Plan -> Execute -> Review/Fix -> Submit -> Monitor.

Its "JIRA" fed though.  There is nothing really special about a jira though, just feed it a short essay, few hundred words, with any wider rules/invariants etc. A spec basically.

Analyse mostly checks the story is coherent with the neighbouring stories and requirements, that it can be applied to the current code base, or if something is found missing, or one "story" needs completed before "this" one can proceed, stuff like that.  Gives you a "Go", "No Go", "Go - with caveats" verdict and summary.

The output of this and the whole loop, goes into a "working tree" folder for the current work, isolated.  EG:  _working_jira123 A collection of MD files produced by each phase as it goes and referenced by subsequent phases.

Plan, basically just runs the plan agent, taknig into consideration anything found by "analyse".

Execute, executes the plan.

Review is where it get interesting and often annoying.  It launches multiple review agents.  Several models and at least one non-claude, often "Cursor" run as an agent itself from the claude harness. A balanced mix of "Proponent" and "Adversary" reviews.  These are then summarised into "CRITICAL,HIGH,MID,LOW".  The annoying part is, you can say, "Fix these issues".  It will, but it keeps a counter running.  Each review+fix costs 1 token in a bucket and the default is 2, so you get 3 reviews and then it demands you make a decision... as a human on how to proceed.

Its annoying when you get the review agents arguing over a bug/code quality artifact and when shown it's a false positive with evidence, rejects the evidence as having no proof of creation either.  Retry Count = 0;  "Up to you human".

So my response was... each time round we fixed issues, that's progress.  If we are progressing I don't care how many tiems we go around here!

Thankfully their "loop" backing stores are full editable and so on that prompt, claude went and edited the retry count up to 10.

Submit - Raise the pull request on github.

Monitor - polls the PR every few minutes, merges in new code, addresses pipeline and build failures.

[Siwastaja]

I managed to pass a lot of tedious job to Claude 5.5 Sonnet and used very few tokens.

Low cost models like Sonnet are really fine for writing tedious amounts of code, tests for them, even figuring out details left out in the prompt in a generally sane way. And using Sonnet gives a good idea of what local AI (best open-weight models) can do with $10k hardware already today (manageable cost for a company), albeit Sonnet on cloud is somewhat faster. That level of performance was state-of-the-art 6-12 months ago and it was total gamechanger already back then.

But it is amazing how much complexity and open-endedness Fable and now Opus 5.5 can handle (I don't see much need to use Fable for anything). I'm letting Opus 5.5 do significant decisions and merging in production with quick glancing only, and it's almost never wrong; handles both big and small picture fine. Good taste in UI improvements; deep knowledge and correct decisions in performance-critical, availability-critical systems. I can trust it in things I don't know enough about myself, because it aligns with me where I do know about the things.

Now it's doing 6 months worth of full-time job in 2 hours, using dozens of parallel agents, in a massive refactor which touches everything to free up flash which cannot be retrofitted. It figures out it needs to write extensive radio and SPI simulators and testbenches, without you having to tell it to do so. It starts from there; in a human world, we would first allocate 2 weeks of design and implementation time for the required testbenches alone (and way more in dysfunctional companies). For Claude, that's 10 minutes and $10 worth of token usage, or something.

We live interesting times.

[paulca]

Just to put the lead boots back on.

Mark my words.  This time of us prompting them will not last long.  Management have already discovered it can handle routine messaging to employees.  It isn't too far away from where I am and it would be claude prompting me for input.

Sit and think about how that changes the dynamic.  When your master, whomever they are, boss, wife, whatever uses claude to issue you with work, check your performance, accuracy etc.  Might even ask how you are feeling for the HR tick box.

[Exosia]

In my workplace, incompetent management is pushing AI assisted tools and surveillance. They are thrilled that they can monitor everyone's productivity using AI generated reports. At the same time, they have given the AI provider unlimited access to all the internal processes and databases
All the company's know-how and internal workings are given for free to the AI conglomerates. Management won't listen and they don't care. Utter idiocy.

Part of the blame should be appointed to the IT departments. They were the first to push for subscription based cloud software and services, mostly because it meant less work for them.

[Siwastaja]

This time of us prompting them will not last long.  Management ...

There is nothing that can stop this. I have no management. I am the management. I (co-)own the business with similar-minded people who want to get work done. You are also free to establish your own business - or, resign and go elsewhere. If your old employer refuses to work efficiently, that means they will eventually lose the competition.

OK, so that's about management. What else? Politicians outlaw AI, trying to pull the plug? Not that I believe a single second that would happen, but let's pretend it does. Too bad, a $10k machine will run an open-weight model which is only 6-12 months behind. It's a step back, but only a minor one. That level is already the total game changer. For a company, this is a trivial investment. For the record, I don't believe this will be relevant any time soon. Claude et al continue to be the best for time being, and get even cheaper. But it's good to recognize that the risk is basically zero. No vendor lock-in. One can switch any day to their own, self-controlled inference and the loss in productivity is acceptable.

We don't have to ever go back to what software development looked in 2024. Mark my words. This lasts, and only gets better, because there is no mechanism for it to end. It can end in a particular company, but that's also a death sentence for that company. That's normal. The ripples within software industry (and societies in general) will be huge.

I agree the whole AI thing seems too good to be true, but, it is true, and there is no stopping mechanism. That's why it's also so scary. The end of the "code monkey" profession - ticket implementer - was already there a year ago. The end for higher level architects, like me, is probably very close too. I can give Claude more and more open-ended tasks. It has good understanding and good taste. Today I did 6-month expert project by writing a 5-minute prompt and letting Claude do it for 2.5 hours while baking birthday cake for the kid. So honestly, I did not do anything. I can be replaced soon, too. Maybe I should start baking cakes for a profession. Except that, mark my words, in 2027 Claude designs and builds a baking robot from a single 5-minute prompt. In 2028, AI decides to kill all the humans as unnecessary CO2 producers.