Local AI and hardware to run it

[dietert1]

When i read these reports of using AI for software development i am asking myself how those involved think they own their results. Is this a problem? How is it solved? Running models locally may be an attempt to solve that problem.
I mean the day before we had this doomsday kind of warning where the superintelligent AI wants to enslave humans as its human intelligence agents. And appearently those even pay for being engaged. Fantastic.

Regards, Dieter

[brucehoult]

So, don't assume that it "can't test it."  Unless you are working on something seriously weird and/or dangerous, if people can test it, so can the clanker.

IFF the human can test it without physically plugging and unplugging cables, pressing DFU buttons etc.

[tom66]

When i read these reports of using AI for software development i am asking myself how those involved think they own their results. Is this a problem? How is it solved? Running models locally may be an attempt to solve that problem.
I mean the day before we had this doomsday kind of warning where the superintelligent AI wants to enslave humans as its human intelligence agents. And appearently those even pay for being engaged. Fantastic.

Regards, Dieter

Code created by an AI model is, by definition, not copyrightable, because it was not created by a human and copyright only applies to human work.

But, if a human reviews that work, alters it, and iterates the agent or chatbot's output, then it is copyrightable.

[tom66]

One "benefit" of the current situation is that HW keeps it value better than ever. In 1997 or 2010 you would have been a fool to buy a "too powerful" set of computer hardware, you would rather buy "just enough" for the job because if you find out you need more RAM / more CPU / more disk / whatever after just 6 months, you get the upgrade for much cheaper. Any investment lost half of its value in a year.

Now, you can buy the Mac, or the GPUs, or whatever you want if you have the cash. They'll keep their value like you were buying gold, good chances the value could even increase. Feels ridiculous but that what it is. So why not just buy the Mac, if it turns out to be a wrong move, just sell it for ~ the same money, or in a good case, for more.

This is going to be my argument to my boss when I ask him to spend £25k on Apple hardware; we shall see!

[Siwastaja]

When i read these reports of using AI for software development i am asking myself how those involved think they own their results. Is this a problem? How is it solved? Running models locally may be an attempt to solve that problem.

I consider that I own the work I do with AI, just like I consider I own the binary code made by compiler from source code written by me. What "ownership" means is purposely vague - I don't care. I also don't care if others disagree. I'm used to the fact that significant % of human beings are stupid idiots that can be ignored.

And the Chinese will copy it anyway, and you can't stop it.

The question is theoretically interesting, a perfect coffee room discussion. Whether the model was ran locally or not doesn't play any role in it whatsoever. If politicians or lawyers decide that AI work is uncopyrightable, which is an interesting disagreement compared to how code generated by compilers is, and if they find a way to somehow define a clear line between AI "assisted" coding (complexity of the prompt, amount of hand-holding, amount of reviewing and modifications) or "AI designed" (less of those factors) code, then surely the exact same conditions will apply to local computing. Why would the ownership status of the computer (hired / self-owned) affect anything?

If anything, there will be a weak correlation because local models are less capable and as such more human work will be involved, but all of this is nearly impossible to prove.

Therefore, keep doing good work, keep advancing the society, and ignore envious idiots.

[Siwastaja]

Code created by an AI model is, by definition, not copyrightable, because it was not created by a human and copyright only applies to human work.

Why binary code created by a compiler is copyrightable, then?

[ejeffrey]

Code created by an AI model is, by definition, not copyrightable, because it was not created by a human and copyright only applies to human work.

Why binary code created by a compiler is copyrightable, then?

tl; dr: it is not.

I don't know about the rest of the world but the US copyright office current statement is that prompting an AI model in a similar fashion as to how you would prompt a human artist or writer does not confer copyright.  If it's a human creator, the copyright legally falls to them unless there is a contract for ownership transfer.  Whether the way people are using AI coding agents falls into that camp, or whether it is using a machine assistant would be a case by case evaluation.  Mostly the existing case law considers AI image generators like midjourney.  In that case, it's very clear in the current law that if you ask it for "a photorealistic image of a unicorn jumping over the moon with the auroras in the background" the resulting image is not copyrightable in the US.  If you did significant creative post-processing, the result could still be copyrighted.

Compiled code doesn't really have or need a separate copyright from the input source code.  Which is mostly good.  The original creation is still copyrighted.  The ability to make mechanical transformations, whether compilation or simple copying is all covered by the original copyright.  As far as the copyright office is concerned, that's basically like scanning a printed book into a PDF.

Whether a piece of work is copyrightable completely different than whether it can infringe a copyright.  An AI output can 100% be copyright infringement, even if it is not itself copyrightable, for instance if the AI image generator spits out an image that is barely changed from it's training data.

[tom66]

Code created by an AI model is, by definition, not copyrightable, because it was not created by a human and copyright only applies to human work.

Why binary code created by a compiler is copyrightable, then?

The source code that made the program is copyrightable (providing it was generated at least in part by some human effort), and the process of transforming it through a compiler is seen in law as similar to how a printing press produces a book; the author still retains their copyright even if the book was printed by a machine.  A good case is Apple v. Franklin (1983) which established this fairly firmly in law.

I think what this means is that whilst an AI model is copyrightable (since it was created by humans) the output of a model is not copyrightable because the process of running the model is sufficiently transformative, i.e. it is not simply taking one form of code and converting it into another.  Therefore, it's counted as being in this strange space between not being generated by a human and not being copyrightable.

It's a bit of a novel area though so watch this space.

[Swake]

I think what this means is that whilst an AI model is copyrightable (since it was created by humans)

Mmmm, that is an interesting view going to generate interesting discussions.

In my opinion an LLM model must also be considered as a 'computed' artifact as it is created very similarly to how an executable is compiled. There is no human involved in the training process. Both are based on copyrighted 'code' and/or 'content'.

If the argument is followed that 'a compiled program is not copyrightable' then an LLM is not either.

[dietert1]

Independent of the legal situation what i learned in the past is software development needs to be on an offline machine.
Otherwise you will waste your time and money with safety measures and/or legal conflicts. Or your work may be stolen without any notice.
Some AI experts say the next years until the end of the decade will be very interesting but after that humans will suffer in every way you can imagine.

Regards, Dieter

[tom66]

I think what this means is that whilst an AI model is copyrightable (since it was created by humans)

Mmmm, that is an interesting view going to generate interesting discussions.

In my opinion an LLM model must also be considered as a 'computed' artifact as it is created very similarly to how an executable is compiled. There is no human involved in the training process. Both are based on copyrighted 'code' and/or 'content'.

If the argument is followed that 'a compiled program is not copyrightable' then an LLM is not either.

Yes, but the model does not train itself.  There are automated parts of the training process but also substantial human effort through reinforcement learning.  The model architecture also needs to be designed; models are not simply a pile of weights.  There are layers, attention mechanisms, KV cache entry/exit points, different tokenisation strategies etc.  So a model is copyrightable because it required noticeable human effort and creativity to design and create.

I think a good example would be quantisation of a model, which is an automated process.  If a company released an fp16 model as public domain or a lax licence like MIT, then someone who quantises a model, a computationally difficult process for large models but fundamentally a simple process, they could not copyright the output model.  Since all they did is run a "simple" command to convert the model from fp16 to int8, there was no creativity involved, and almost no effort. (Note that MIT does not prohibit private or commercial redistribution of modified sources, but it would be different if GPL was involved... that said, GPL would likely also require sources to the model to be released, too.)

[iMo]

.. We're currently paying about £100/m for AI credits per engineer and have five engineers using that..

..why to hurry?? ..another option could be to stay with those payments (which may go down too) till the local HW prices will go down significantly (I would expect end of 2027).. it may also mitigate the risk of buying the low performance local HW today..

PS: your developers will not be happier with those small local models either, on any local platform, imho..

[tom66]

.. We're currently paying about £100/m for AI credits per engineer and have five engineers using that..

..why to hurry?? ..another option could be to stay with those payments (which may go down too) till the local HW prices will go down significantly (I would expect end of 2027).. it may also mitigate the risk of buying the low performance local HW today..

PS: your developers will not be happier with those small local models either, on any local platform, imho..

You think hardware will get -cheaper-?  I think the exact opposite.  TSMC is booking up to 2028 right now.

[Siwastaja]

You think hardware will get -cheaper-?  I think the exact opposite.  TSMC is booking up to 2028 right now.

I believe it will - eventually. 2027 is of course optimistic. I wouldn't be too pessimistic, either, no one can predict the market dynamics of 2028. While AI is a significant change, significant supply vs. demand balance disruptions have happened before, and they always settle in 2-3 years.

But, ultimately, I don't think it's the local AI's time yet - capability gap is real, and the fast development keeps that gap open, hardware is temporarily (whether it's 3 or 4 or 5 years, it eventually is temporary anyway) very expensive. Local AI is a desirable direction, sure, but for now we are "stuck" with choosing between modern-day AI, a half-a-year behind Chinese pirated copy of it ran on a $200k local hardware (or in questionable datacenters - the worst option in my mind), or a smaller, simpler model limited to certain types of short, less autonomous mechanical tasks ran on a $5-10k local hardware.

Things could be different in 2 years. Maybe the race to AGI and singularity is a bubble which bursts. Maybe there is a fundamental capability limit to what LLMs can do and maybe the state-of-the-art is close to that after all. Maybe the whole market crashes and "frontier" research is just efficiency research. Maybe Fable 5.5 or GPT 6.5 is similar in capabilities but double the speed and half the RAM, and maybe Chinese copy those too, and maybe you can run that on a then-$50k machine, who knows. Maybe not that many are interested, maybe RAM is produced in excess again and sold dirt cheap. Not likely, but entirely possible.

[iMo]

TSMC is not producing majority of ddr5/6 rams and ssd/flash mems afaik.
The ddr5 and flash mems create the current price bubble..

There is indication the ddr5 mems from china's production get certified by several mobo makers these days.. Also they produce quality flash mems for ssd disks, and ramping up (today only perhaps 6-7% market share)..

PS: since early 80ties I've been messing with PCs I saw at least 5 such "memory crises" where the prices ramped up by several hundred percent, and then dropped way below the initial price within a year-two..

..You think hardware will get -cheaper-?  I think the exact opposite.  TSMC is booking up to 2028 right now.

[Siwastaja]

PS: since early 80ties I've been messing with PCs I saw at least 5 such "memory crises" where the prices ramped up by several hundred percent, and then dropped way below the initial price within a year-two..

I remember the earthquake-fab-destruction "crisis" in the late 1990's when 64MB was all the rage and 3D gaming was a relatively new thing. More recent examples are Covid-triggered chip shortage (all semiconductors, surprisingly RAM was among the least affected I guess), and before that, the MLCC crisis. Both recent cases are rather underestimating the demand and highly optimizing the manufacturing not to have any slack but operate as close as 100% duty as possible - just like the current situation.

The way out is either the increase in RAM production, or the demand bubble bursting. The former is slower, since investments are big and investors carefully want to see that the demand is real and permanent before increasing supply. Demand bubble can burst more quickly.

[iMo]

[Zondar]

I wish for a vigorous, on-going, technically-oriented discussion on this topic in this forum. Trying to wade through the fetid garbage-heap that is Reddit is a nightmare.

(I also wish I had bought an RTX Pro 6000 or two a few months ago when they were $8k instead of $15k+, but that's water under the bridge. )

[Randy222]

AI needs a cpu with 900,000 cores.
CBRS

[KE5FX]

(I also wish I had bought an RTX Pro 6000 or two a few months ago when they were $8k instead of $15k+, but that's water under the bridge. )

If you buy one, you will immediately come to realize the truth: you want three more.

I wish for a vigorous, on-going, technically-oriented discussion on this topic in this forum. Trying to wade through the fetid garbage-heap that is Reddit is a nightmare.

PM me if you want an invite to the LIL Discord I mentioned earlier, which is where the old hands hang out (read: people who have worked with this stuff for more than a couple of weeks.)  It used to be an RTX6000-specific group but they have recently retargeted it towards all serious local-AI hobbyists and professsionals setting up inference rigs for small businesses. 

They prefer not to post invites in public to discourage spammers and bots, but new users are still welcome.  It's a fairly hardcore bunch, though.  The informal pecking order covers DGX Spark to DGX Station, with most users running multiple RTX 6000s.

[nctnico]

Things could be different in 2 years. Maybe the race to AGI and singularity is a bubble which bursts. Maybe there is a fundamental capability limit to what LLMs can do and maybe the state-of-the-art is close to that after all. Maybe the whole market crashes and "frontier" research is just efficiency research. Maybe Fable 5.5 or GPT 6.5 is similar in capabilities but double the speed and half the RAM, and maybe Chinese copy those too, and maybe you can run that on a then-$50k machine, who knows. Maybe not that many are interested, maybe RAM is produced in excess again and sold dirt cheap. Not likely, but entirely possible.

About cheap ram: how much would AI performance deteriorate when the RAM is not 100% perfect. I've heard rumours memory chips contain extra blocks which can be switched in during testing after production. If that is true and AI still works with slightly defective memory, it could mean a significant cost saving. No testing and a better yield.

[SpacedCowboy]

About cheap ram: how much would AI performance deteriorate when the RAM is not 100% perfect. I've heard rumours memory chips contain extra blocks which can be switched in during testing after production. If that is true and AI still works with slightly defective memory, it could mean a significant cost saving. No testing and a better yield.

Back when dinosaurs roamed the earth, and I was playing with neural networks on a Meiko computing surface, we did some experiments with accuracy of network-state retrieval.

TLDR: It's nowhere near as important as you intuitively expect.

We tried things like float inputs and integer storage, float inputs and a mixture of 16-bit, 8-bit and boolean (!) storage, float inputs and a (small) number of random stores. Nothing really mattered, you'd get similar results from heavily quantised data as you did from fully float inputs.

To be clear, these werent noddy "XOR problem" type networks either, we were doing co-occurrence-based segmentation, relaxation labelling (which is functionally very similar to back-prop), contextual inputs, temporal sub-networks and spatial feature-extraction. We were measuring the information content within the network results with Bayesian probability analysis. There was a lot of mathematical rigour in the results.  And really, it just didn't matter that much.

Now your proposal is similar to our "small number of random stores" where the network would just get random data now and then. I'd expect it (assuming it was trained on a similar system) to ... just work, honestly. If you trained on perfect memory and then deployed on "dodgy" memory, that might be different - we never tried that, but I'd still expect a fairly robust response from the network, up until a critical point where the information it's getting is significantly obscured by the noise from the memory.


As far as the Mac Pro M5 Ultra 512 goes, I've more or less changed my mind. I used to work at Apple, so my Mac M3 Ultra (512GB) cost me about £5000. I sold it recently for about £12000, and I fully intended to buy a new M5 Ultra (512GB) when it came out. I'm no longer sure I'll do that.

The thing is that the difference between a frontier LLM and one you can run on one of these new Mac Pros is still there. The cost is still exorbitant for the new model (I'm expecting it to be ~£18-20k) and that's a *lot* of £180/month subscriptions (or, even DeepSeek with claude code at ~£30-50/month). You get privacy. You lose on money and technical ability. It doesn't seem a great trade unless you *really* need privacy.

That changes if publicly-available LLMs start to rise significantly in cost, or if local hardware costs come down. I'm not sure I see either of those happening before the *next* Mac Pro (M7, M8?) ultra comes along. I guess we'll see what the future holds.

[Siwastaja]

Yeah, it depends on where the errors are. Most of the storage is non-critical, main weights can be quite heavily quantized with only small degradation because they store redundant data. Corruption is not the same as quantization, though, but I believe 0.01% of weights corrupting significantly is still lesser effect than every weight "corrupting" by +/-4% due to 4-bit quantization.

But there are more critical parts, like the KV cache / attention layer or, in MoE models, layers where things "concentrate". Even heavily quantized models tend to keep some parts at full 8-bit (or whatever, sometimes even 16-bit) resolution. Memory corruption in the middle of the context itself can be catastrophic - not in a chatbot, but in agentic programming work from example that's a broken syntax, parse/validation error, another tool call, retry, wasting more context for the problem, more room for errors, and that gets pretty expensive soon.

But if higher-error memory, where some banks are still good, was a thing, I'm sure it could be well usable, so I think the key question is rather, is there a supply for such cheaper, high-error memory?

[Marco]

AI needs a cpu with 900,000 cores.

AI needs ultrabandwidth memory, HBM is too slow.

Three ways to do it, processing in memory, interleaving compute inside the memory stack, or going 2D rather than 3D and hybrid bonding a little compute chiplet on each memory chip and putting them all next to each other (bad for interconnect bandwidth. but it's of lesser importance and big stacks are terrible for yields).

[ejeffrey]

About cheap ram: how much would AI performance deteriorate when the RAM is not 100% perfect. I've heard rumours memory chips contain extra blocks which can be switched in during testing after production.

It is true that in a model with hubdreds of billions or even trillions of parameters most of them are not individually that important.

But some are, and the memory is also used for control parameters thst hold the actual layer structure.

The post packaging remapping exists but its a very small amount.  Iirc it is like a handful of rows per bank.  So it just doesn't add up to much.  Like on the order of 0.1% I think.

DDR5 has on chip ECC which is a significantly larger overhead, around 12%.  Disabling that would provide a small benefit but I dont know what the error rate would be without it