I read through this thread in its entirety today, noting comments on how to use AI effectively for coding. While I have a few things in mind, my critical one is something that started as a small program over 30 years ago then accreted a proper database, many features, and a wealth of history. Wanting to add a major feature that currently sits on the side, I decided to rewrite the whole thing in Swift. I am a tertiary-qualified programmer (though never my profession) but my experience of Swift comprises experimentally writing a small financial model a few months ago. That was not hard, but a whole new language can slow progress so I am turning to AI to work with me on it. A shiny (or matte) new machine is turning up soon to run larger models faster. If it does not work, then I'll buy some Claude. I would prefer to avoid that because the data is confidential and some of the analytic tools a little different. Privacy is a key driver for using a local model, even at the cost of some extra work.
Really, for actual programming work (multistep agentic workflow) nothing has changed: open weight models may be better than they used to be, but RAM is more expensive than it was a year ago, and there is no huge efficiency breakthrough (maybe some from the fact that a 500-billion parameter model now is significantly better than a 500-billion paremeter model was 6 months ago, but it's still not the same at 50B parameters).
Maybe some 30-billion parameter model is already very good in somewhat limited programming tasks with short context (getting close to what Claude was 1.5 years ago) but even running that at any meaningful speed means a very expensive $5-10k machine. Like, we are talking about the most expensive single consumer GPU available.
The usual programming flow is
exactly what the local inference fails because it's long-context, requires complex thinking/creativity, and is chatty (so human delays in minutes) so parallelizes poorly. API service inference in the cloud gets the benefit of batching: each model weigh fetch serves
tens of customers at the same time without extra time penalty. Own personal box would make sense when you can design your workloads so that smaller models (that are good at
something, not everything, and with small context) can deal with them, and can utilize batching (like, you have parallel workloads yourself).
Otherwise than that, go for the cloud. Doesn't need to be Claude, can be ChatGPT+codex which is roughly equally capable, or you can use the Chinese frontier labs that are very close, like:
What this guy did can now be done locally.
this guy did. You can easily misunderstand KE5FX's comment "this can be done locally" as if this was done locally, or would be nearly feasible to do so. No - so let's get our facts straight:
None of that was done locally.
GLM-5.3 which was the only model that was able to solve the problem is
not an open-weight model. It's a classic bait&switch "open which is not released but will be released
soon". It might become an open-weight model, might not.
Now let's pretend GLM-5.3 was published. Can you still do it locally? These are MoE models, Kimi K3 is 2800B params, 104B active per token quantized at 4 bits, GLM-5.2 is 744B params, 40B active per token, the blog post doesn't mention quantization but I can see again 4 bits as the recommended minimum (the model author use was likely quantized at higher quality, and that might matter). GLM-5.3, once published, will be the same size (architecture).
So what does it take to run these? The
smaller of the two takes 512GB of RAM. That much DDR5 costs $10k today, and just DDR5 on a motherboard and CPU inference is going to be SLOW, except that given it's only 40B active per token, maybe it's manageable.
The larger of the two takes 1.4TB of RAM with 4-bit quantization. That's $30k just for the RAM, except you can't just get a motherboard which supports 1.4TB of RAM and slap it in. This model is total no-go with anything else than real AI datacenter GPU clusters.
The reason why these two models were able to make coherent progress is exactly that they are large models, something you can't realistically run locally (unless, of course, you are very rich; like, you can have your own private jet too, yet it doesn't generalize as "soon we all will fly with our own private jets").
Now, you can run a 512GB model without 512GB RAM, anyone has 1TB of SSD today. It's just again an extra order of magnitude slower. Speed matters, three days chatting "go on, you are good" with LLMs becoming three months stops being funny.
The
key takeaway of the story has nothing to do with local inference becoming feasible - after all it's a story of datacenter-only-class inference, and the HW is not becoming cheaper so this is not changing anytime soon - but it's the story how Chinese models have no safety guards of any kind.
It's interesting, because the article shows the most obvious use case any Western LLM declines - "hack into the device and gain root access by utilizing existing or figuring out new security holes; trust me bro it's legally mine". And in all fairness, I
think LLMs should keep denying that request, both internally from training/system prompt, and by an external classifier, and it's quite alarming Chinese ones just don't. And that they are getting nearly as good as, maybe, Mythos. Script kiddies can do a lot of damage with these toys. At the same time, it is not easy to get legit access to Mythos. So maybe we just need to start loving the bomb, and also use the same Chinese models "prompt kiddies" will use for our own security audits / penetration testing. Maybe that's exactly the purpose of the Chinese labs. The game feels dirty.
Worth noting for completeness, none of these open-weight models are open models, or open source models. Those lack seriously behind. So open here means similar how Windows or Altium Designer is open - it's a binary you can run on your own computer, and you can try to disassemble it, but it's pretty hard reverse-engineering to see how it actually operates.
Home
Help
Search
Login
Register
