Code red at OpenAI as it ‘pours money down a black hole’

[perieanuo]

There is a real risk someone will crack how to run good-enough AI/LLM much faster and with much less RAM and that will mean you can run the LLMs on your own hardware instead of paying a subscription.

ok you got the sw running but the data pool is not on your hands so the agent will parse my 16TB data? nogo

[Berni]

There is a real risk someone will crack how to run good-enough AI/LLM much faster and with much less RAM and that will mean you can run the LLMs on your own hardware instead of paying a subscription.

ok you got the sw running but the data pool is not on your hands so the agent will parse my 16TB data? nogo

That's exactly where it might make sense to consider local AI inference.

Feeding this much data into a model using the API of the big popular AI providers (Google, Anthropic, OpenAI) will have them sending you a bill for around $1 000 000 to $100 000 000 in API usage costs(depending on how smart of a model you need). Tho that is just the input token cost, if you also want to output 16TB worth of tokens back out of a model that might cost you as much as 500 million USD.

Not saying that doing this local would be cheap either. Just the amount of electricity you would burn for this is for sure in the 10s of thousands of dollars minimum.

[ejeffrey]

yep, there's a lot of misinformation or assumptions that local AI must be absolutely terrible because the big players need such huge datacenters to power LLMs.
It's really quite good and keeps getting better.

RAM&HDD shortage and price explosion alone is proof that high-quality, fast local AI is impossible in large scale as we speak.

Impossible in large scale does not mean impossible.  If 900M people suddenly took up FPS gaming and bought a high end GPU in one year that would also cause crazy shortages.

It definitely seems like the open models are behind the commercial models in efficiency and the largest models of both are too big to run efficiently on any sensible local accelerator (requiring up to 1 TB of aggregate GPU memory).  But it's not crazy to imagine that we are pretty close.  The largest of the newly announced google open source models can run on a high end consumer GPU if quantized down to 4 bit.  Right now Nvidia and AMD are not keen to increase GDDR on consumer GPUs because they want to sell $20,000 H100s not $2000 RTX5090s, but it's sensible to imagine that in a year or two, either Nvidia might be able to release high RAM GPUs, or one of the companies making dedicated AI accelerators might make a consumer version with say 64 GB or 128 GB of memory.  Even if it's not the absolute fastest in terms of FLOPs.  That's still not enough to run last years largest commecial coding models, but it's getting there and with the recent work in model compression it might make it.  A lot depends on whether the model compression tilts the cost-benefit ratio towards smaller or larger models.

[SiliconWizard]

The situation with "AI" is insane, but beyond that, it shows the obvious: the whole world relying on a very few number of foundries for semiconductors (TSMC itself being clearly a bottleneck) and even fewer RAM manufacturers is absolutely nuts.

Yes, foundries are large investments, but come on. We waste hunreds of billions on useless crap, nobody can convince me this money could not be better invested in what matters: power plants and foundries. (I'm only talking about industrial needs here. If you were going to say: what matters is food and water, I won't disagree either.)

[Marco]

requiring up to 1 TB of aggregate GPU memory

For a single user, generation can run on CPU and prompt processing can be done on the GPU layer by layer, streaming in the parameters.

[Berni]

All large changes in demand cause market instability.

Half a decade ago it was also impossible to buy a performant graphics card at a sensible price. Except back then it was because of the cryptocurrency hype where we had crypto miners buying up however many GPUs the factories could produce. It was very profitable at first, but then as the number of miners exploded the mining rewards got spread so thin that nobody was making any money anymore and suddenly you could buy whole pallets of used GPUs for a fraction of what they cost new.

Unfortunately cryptomining is mostly compute bottlenecked and doesn't need much VRAM, so the cards that have been used for that are usually not good for AI, while the prices on used cards that do have a lot of fast VRAM have went up 2x or 3x because people are buying them up for local AI.

As for running a huge 1000B parameter model locally. Yes you can do it but it is not cheap and is usually slow. Typically models this large are MOE models where only the relevant "experts" in the model are ran so that massively reduces the compute and bandwidth requirements per token(at the price of making the model slightly dumber). You can do these on CPU, but you still need the 1TB of RAM (that is anything but cheap these days) and something to put it in. This is usually either a huge server motherboard with 24 RAM slots, or many consumer motherboards each holding 128GB of RAM and coordinating over the network. Using the new Mac Minis is also popular due to their unified memory architecture. But if you want to run it at speed you do need 10 to 50 graphics cards working together. Tho getting the most performance out of it involves processing multiple users prompts in parallel.

The worrying part is indeed the chip fabs. We got mostly just TSMC in Taiwan propping up the compute in AI while Korea makes most of the RAM needed for it. And with tensions rising all around the world it is not that far fetched that a conflict might flare up in that part of the world too. Then suddenly the whole world is without consumer or datacenter computers.

[Siwastaja]

The situation with "AI" is insane, but beyond that, it shows the obvious: the whole world relying on a very few number of foundries for semiconductors (TSMC itself being clearly a bottleneck) and even fewer RAM manufacturers is absolutely nuts.

Sure, but nothing to do with AI. Even before AI, our daily lives completely depended on computer systems, already running solely on CPUs and RAM from few giants.

It is efficient, but it is also risky.

Around the turn of the millennium, I remember there was already RAM shortage and price peak because nearly all RAM was made in a single factory, affected by an earthquake. That was back when 64MB was nice amount of memory.

[Berni]

Here we go. Anthropic started doing cost saving as well.

To be fair part of it is people abusing their Anthropic subscription, so they closed down the trick. But the timing of it suggests that they are having similar cash problems as OpenAI

[Siwastaja]

Here we go. Anthropic started doing cost saving as well.

To be fair part of it is people abusing their Anthropic subscription, so they closed down the trick. But the timing of it suggests that they are having similar cash problems as OpenAI

Definitely. Enshittification is the biggest risk AI users face. I hope it's rather "these features which are heavy on our resources become pay-per-use", than just making everything shittier for all, even those who are willing to pay.

It's crazy how much value this thing has to offer for $100/month. I'm willing to pay ten times as much, if it needs to be. Then again, pricing of services is difficult. If they just increase the price by 10x, they probably lose more than 90% of the subscriptions. Pay-per-use is hard to predict and thus feels risky to users - I'm not willing to use a service which may cost me $10000 suddenly. Then again, $100 fixed is too good to be true. So, fixed price, but pay-per-feature, then - pay more for premium features? But that's difficult, because the AI part itself is capable, so it's also capable of helping you to create whatever tooling you want, bypassing special feature payments. Like, Anthropic offers some "Claude reviews your github pull requests" feature which costs like $10-20 per pull request, but does anyone use it when you can just launch claude (maybe from fresh context) and ask "review this PR"? For the same $100 or $200 per month which you are paying anyway.

So that means they need to come up with creative ways to lock down expensive patterns. And that will inevitably cause "false positives", enshittify the service for those who do not cause heavy use.

I guess one of the most harmful patterns is trying to put AI to do a large, complex task without supervision. Clearly the models are taught with mechanisms to get "bored" or "tired", you can clearly see this - you can't just say "keep working on this problem until X, Y and Z". It tries approximately 3 things, and then summarizes the results and stops. And it's not about context getting full. Autocompact retains the context, and the behavior is same even with 1M token context. It's some real limiter in the model behavior, which must be trained in. People circumvent this by relauncing with fresh context, or using some master supervisor thing which follows a plan and launches other agents, etc. etc.

I have tried this unsupervised "complex task" pattern twice so far and it simply did not work, AI is clearly not ready for that yet. Maybe in some special cases where the problem is far from novel, and earlier footsteps can be followed - with a clear implementation plan, test plan, and exact numerical outcome. But otherwise, it lacks correct intuition and makes poor choices. And wastes a lot of resources doing work which needs to be mostly ditched later. Therefore, human-in-the-loop is very beneficial - you get what you want, by working together with it. Plus, huge amounts of resources are not wasted doing unnecessary work.

The feeling you can let it do anything for fixed price, only to ditch the results is extremely harmful and that's what they need to prevent.

[Berni]

Exactly, everyone is pricing AI so cheap so that they would win over customers.

Luckily with AI services you can fairly easily establish tiers of service. The cheap plan users get the smaller model. That model is a lot cheaper to run (so one inference server can handle many more users) and at the same time the model is not as smart (so there is incentive to upsell).

But right now the cheap/free customers are the ones that later upgrade to the expensive subscriptions, so they don't want to throw the small dumb models at them. They might use the dumb model and go "This AI is crap" and leave.

As for leaving the AI agents to do work unattended, that's a bad idea for now. Pretty much all of the LLM models we have right now gradually start going insane if they talk to themselves for too long. That's well demonstrated by the experiments where they had an LLM run a small business like a vending machine. Usually it starts off pretty well but then as unusual things pop up inside its context the LLM starts doing unusual things and filling the context with more weird stuff. With each iteration this then gets amplified more and more until and eventually that vending machine LLM is trying to send an email to the FBI.

As for this behavior that it tries a few things and then gives up is likely from how it was trained to work along side a human rather than alone. The giving up is likely intentional to get the human to consider a different approach to the goal, so this ended up with a better success rate (since they use peoples chats as training/testing data)

In any case the money is drying up in the AI bubble and the era of enshitification is coming. Hence why having good open weight models anyone can run will be important (even if you pay someone else to run them for you).

[Siwastaja]

Exactly, everyone is pricing AI so cheap so that they would win over customers.

Luckily with AI services you can fairly easily establish tiers of service. The cheap plan users get the smaller model.

Except that if you want to lure more software developers into using AI, you would not want to show some lower-quality product, but the "real thing".

Quickly glancing over the video, the problem seems to be people wiring that expensive model, designed for real software development work, into routinely reading their viagra spam emails and similar crap. I'm not surprised at all Anthropic does whatever they can to block that use. This seems the most sensible response so far: try to detect harmful usage patterns and block them one by one. I like it - except, maybe one day my usage is considered harmful, either on purpose or accident, and then I will be mad. It's a difficult balance - they need to piss off only those customers that are net-negative AND who are not useful for PR - who will not bring in more customers, and will not transition into beneficial users themselves either - while avoiding pissing off those who are beneficial.

Usually, every road leads to enshittification - services which stay good are rare. But there is a glimpse of hope - vendor lock-in isn't a thing with AI. Say, you code with Claude Code. Absolutely nothing prevents you from subscribing into one of their competitors and start using it instead. All the work is being done on the code, your files. The greatest shortcoming of AI, total lack of longer-term memory and context understanding, is beneficial in this case. Exactly because Claude has to reread the relevant parts of your entire project each session (or depend on small natural language summaries), switching to a different vendor is no different at all! Like, the worst vendor lock-in currently is that you need to rename CLAUDE.md -> AGENTS.md or create a symlink.

You can even literally continue a session with different provider by playing back the discussion log!

Currently: harsh competition -> good service for cheap.

Now, if/when they are able to solve that huge "mid-term memory" gap - it seems they are nowhere close yet - that will mean AI gets much better, but it also means opportunity for vendor lock-in; opportunity for enshittification due to high "cost of switching". In that sense, I pretty much like the current AI suffering from pathological dementia.

[booscrawl]

I'm curious about this.  How do the state of the art local models work on larger code bases?  The "prevailing wisdom" is that local models just can't compete with something like claude code.  The claim is that they can provide stand alone code examples just fine but they can't handle a many file context window and still work well.

But nobody actually explains this other than saying nonsense words like "enterprise grade."  To the extent this is true it's not clear to me if it's even a limitation of the models themselves vs the tooling around the LLM.

You have to try for yourself.

[ejeffrey]

You have to try for yourself.

What a stupid and pointless comment.

Yes I could try for myself and I'm sure the result would be pretty bad.  But I don't know why.  It is presumably some combibation of lack of hardware, lack of software, the best models being private, and me just being a dumbass and not setting it up right.

Software and models and me being a dumbass can be fixed with time.  Hardware is trickier but even then I'm curious what specific requirements are needed to run a single user version or something  competitive with commercial offerings.   

So yeah I want to know how likely is it ther I will be able to run something say 90% as good as Claude code at home in say 3 years.  Especially if I can actually buy memory and GPUs at MSRP by then.

[KE5FX]

I don't think you'll need to wait 3 years, to be honest.  The latest/greatest local models are usable with 100K+ contexts at the Sonnet 4.x level from what I've seen. 

They are not cheap to run, but the shenanigans going on right now with Anthropic and OpenAI make a 90% solution that always performs at 90% look awfully attractive.  The Flowers-for-Algernon routine with Opus 4.6 is getting seriously old.

(Of course, in 3 years Claude should also be much better than it is now, but will any of us still be able to afford it?)

[Berni]

You don't necessarily need to have a powerful computer to make use of open source AI models.

There are lots of providers that will rent you a powerful server by the minute. This lets you get access to some massively powerful machines without breaking the bank.

But for most users it is more convenient to actually rent inference. OpenRouter is a very popular middleman for connecting users to inference providers.
https://openrouter.ai/

Anyone with a lot of GPUs left over can sign up with them to become a inference provider and get money for it. Users can then put some USD credits in their OpenRouter account and just fire off API calls to them as needed. OpenRouter feeds the users API request to one of the available providers and gives them a few cents to compute a response. This way you have access to pretty much all models ran on fast machines and you pay as you go. Quite a few models on here are even free and most of the open source models are really low cost (many of them only costing a fraction of a dollar for 1M output tokens)

[ejeffrey]

I don't think you'll need to wait 3 years, to be honest.  The latest/greatest local models are usable with 100K+ contexts at the Sonnet 4.x level from what I've seen. 

I'm just wondering when I will be able to buy a top of the line GPU for MSRP or whether someone will make a dedicated high performance AI accelerator that is consumer priced.

[KE5FX]

RTX 6000s are still available at prices close to MSRP.  Unfortunately that's already pretty steep, and you will always need N+1 (or worse, N*2) of them to run what you actually want.  

OpenRouter and similar services are likely to be the way to go in the foreseeable future, unless you really want to run everything in house and are willing to pay for it.  I haven't used OpenRouter myself but it's certainly gaining a lot of mindshare.

[Marco]

I'm just wondering when I will be able to buy a top of the line GPU for MSRP or whether someone will make a dedicated high performance AI accelerator that is consumer priced.

Processing is fast enough. What local AI needs is faster flash, not TB/s like cloud inference needs with HBF, a couple hundred GB/s would suffice for running huge models with 4 bit weights. Unfortunately the only interface consumer PCs have with that kind of bandwidth is the memory bus and NVDIMM died.

[Berni]

The reason that LLMs run so well on GPUs is that they typically have by far the fastest memory in a PC

The typical DIMM RAM stick of DDR4 or DDR5 can do around 20 to 50 GB/s over its 64bit data bus. Most consumer motherboards have dual channel memory so that widens the bus to 128bit to give you 40 to 100 GB/s of bandwidth. Sounds pretty damn fast.

However if we take a Nvidia GTX 1070 (obsolete 10 year old GPU with a $380 MSRP back in the day, used value is more like <$90 today) it has GDDR5 on a 256bit bus that does 250 GB/s. That's way faster than system ram today already.

Now if we step up to something more modern say a Nvidia RTX 3090 (6 year old GPU that is still mostly up to date) we get 24GB of GDDR6X on a 384bit bus that is screaming along at 930GB/s. These are very good at running LLMs due to having a lot of fast VRAM, but as a result their prices on the used market jumped from around $600 to over $900 due to people buying them exactly for that.

Going to the pinnacle of consumer GPUs then the RTX 5090 gives you 32GB of GDDR7 on a 512bit bus that does 1790 GB/s. That is an absolutely insane amount of bandwidth. But expect to pay around $3000 to $4000 for that privilege but still less than the workstation variant of it the RTX 6000 (same GPU die as the 5090 but better binned and with way more memory) that goes for around $7000

So the problem is that PCs don't even have any memory interface that can go anywhere near this fast. To get close you need to move over to server platforms such as Threadripper, Epyc, Xeon ...etc where you can get 8 or 12 channel memory. So on the low end of having 8 channel DDR4 you could get to ~250 GB/s. Going fancier with 12 channel DDR5 you could get ~450 GB/s and if you move to dual socket 12 channel DDR5 you can get ~900 GB/s (if you can get the model split properly). But at this point you have a PC that costs around $10000 just for the 2 CPUs then a $5000 motherboard on top and around $5000 to buy 24 sticks of RAM to fill all those memory slots. This thing will also likely consume >300W at idle and more like 1000W when thinking hard.

That being said we have smaller LLM models that still perform great and those run on a decent gaming PC just fine. Running the giant 1TB sized models locally will not be cheap for at least the next 5 years if not 10 years.

[Siwastaja]

The reason that LLMs run so well on GPUs is that they typically have by far the fastest memory in a PC

... the reason why small LLMs run well. A new term: SLLM - small large language model!

The thing is, those models most people prefer for "deep thinking" and complex tasks are 10-30x too large to fit even in expensive side of consumer GPUs. And once they don't fit, the speed benefit is gone. Of course you can run them even from SSD where they easily fit - but it's extremely slow.

So it's a very harsh speed-quality tradeoff most users don't want to take, which is why local AI still isn't really a thing. We are all hoping for the best, of course.

[BadeBhaiya]

Code: [Select]

Qwen3.5-122B-A10B-Uncensored-HauhauCS-Aggressive-Q4_K_P.gguf

May I ask what kind of hardware you are using to run a 122B Q4 model locally? And may I also ask, what are you doing with the "uncensored" model ?

So Google is intentionally burning the AI market to the ground with unsustainable pricing. Once all the competition dies in the fire then they are left standing as the leading AI provider who can then do whatever it wants. So far Anthrophic is holding out with charging high price for their Claude API because their models are very good at programming tasks. But if they slip up and fall behind Google that will kill them too.

We must also remember, Google is the only company who actually makes the chips that they use to train their models on, that is their TPUs. OpenAI, Anthopic and everyone else depend on nVIDIA to give them the very expensive compute that they need. Vertical integration is serving Google very well, they are able to achieve power efficiency beyond what most nVIDIA GPU companies can dream of. Its a wonder why we haven't seen more hardware focused LLM startups and innovations. Hardware too boring for sillicon valley engineers?
I would love to get my hands on some TPUs but alas google does not plan on ever selling them. A single TPU node would solve most of my LLM needs if I can run a 100B+ model locally. Of course, I would find no software to run on it...

[Berni]

Yes the flagship models of the AI providers are massive. They don't tell us how big but safe to say they are above 1000B parameters.

However those are not the models that you usually get to use on places like ChatGPT. The models that get served to the masses for free or at very cheap prices are usually in the range of 100B to 1000B (or even smaller for the really speedy ones). Even so most of the larger ones are MOE architectures (so most of the time only ~10% of relevant expert parameters actually run at a time, massively reduces bandwidth requierments). They have to serve these models at a massive scale so the models have to be reasonably sized to keep the cost of that down. Yes you can still get to use the giant flagship models but they are so expensive that a single prompt might cost you $10 to $100. Only people who really need it are prepared to pay for that.

Additionally there is a limit to how parallelizable AI inference is. So even these massive datacenter GPUs can't actually run large models very fast (yes faster than what consumer GPUs can do, but only by a few X). So most of that these flagship giant models are used for is to train smaller faster models. The giant model doesn't need to be fast in order to generate training examples while these examples are very valuable to teach a smaller model to act closer to the abilities of the big model.

For running LLMs really really fast there are purpose made chips like these: https://groq.com/lpu-architecture
These chips keep memory and compute on the same chip to solve the bandwidth problem. However they don't have a lot of memory so many of these need to coordinate together to run any sizable model. So it is only a viable datacenter solution and even then they scale best for the more reasonable sized models due to the needed coordination. GPUs are still nice because they are more flexible, hence faster time to market.

Not saying that everyone should run local LLMs. To the contrary, just use ChatGPT or Claude or whatever and be happy with it. If you do want local models just rent the inference from someone else. Do NOT go building a massive computer just to run LLMs. Makes more sense if you are a company and must keep your confidential data in house.

Important part is that open source LLMs provide a good market counter balance. It tears the monopoly on AI out of the hands of giant corporations. They are forced to compete against alternative open source AI providers and so can't enshitify their own AI services as much (since their customers can move to alternative options)

[KE5FX]

May I ask what kind of hardware you are using to run a 122B Q4 model locally?

Currently using an RTX 6000 Pro Blackwell, which amounts to a 5090 with 96 GB rather than 32 GB.  They can be found in the US $7K-$10K range. 

Problem is, as soon as you buy one you will want 4, then 8, and before long you're talking about real money.   Just as there were a lot of models you couldn't run on a smaller board, there will still be almost as many you can't run on a single RTX 6000.

And may I also ask, what are you doing with the "uncensored" model ?

The default state should be uncensored.  The question to ask is why everyone else gets a censored model.

An uncensored model means you don't have to wonder if the reply is being censored.  It's nobody else's business if you want to know what happened at Tiananmen Square in 1989, or what popular cartoon character most closely resembles Xi Jinping.  Likewise, if you have a personal problem that falls into a category where the model was conditioned to refuse answers, not because the information is unreliable but because it runs afoul of someone else's religious or political stance.  Or perhaps you object to someone else deciding what you're allowed to learn about drugs, weapons, or other forms of contraband.  Or maybe you just don't want to hear HAL's opinion on your gooning preferences. 

A good example of what I'm talking about is here.  I wanted to explore some possible outcomes in the Persian Gulf conflict, so I posed the same question to all three major providers.  ChatGPT and Claude gave decent answers, while Google's Gemini 3.1 Pro model absolutely refused.  "Sorry, I can't discuss that."  How long before the other providers adopt the same attitude?

You can never back down from fighting censorship, because they'll never go away happy. 

Its a wonder why we haven't seen more hardware focused LLM startups and innovations. Hardware too boring for sillicon valley engineers?

High-bandwidth DRAM is what always limits your ambition when it comes to inference.  There is just no way around that at the present time. 

As a thought exercise, consider replacing a single RTX 6000 card with FPGA boards in a rack.  You will need 100 expensive boards with a gigabyte of fast memory on each, networked in a fully-connected ~2 TB/second mesh.  Few people without VC backing are going to build something like that in their garage, and SV investors are largely allergic to hardware.  Everybody takes it for granted.

[negativ3]

Tho there is opportunity for a big speed boost if we were to develop a AI chip that is 128GB of DRAM stacked on top of a compute die and connected with an absurdly wide 65536 bit bus for ridiculous bandwidth in the 10s of TB/s

I find it very curious not every DRAM and AI chip company is rushing towards developing hybrid bonded DRAM/flash. You could have 2 order of magnitude more connections than that. Computation has become irrelevant, the thermal problems of hybrid bonding for the bottom logic are far less relevant than bandwidth.

The dream team would be hybrid bonding with NEO's 3D DRAM. It tries to produce DRAM cells with 3D flash type processing. Could very well not work though, the capacitance for 1T/0C is tiny and deep layer growth inside tiny deep holes to grow more capacitance in complex structures very optimistic.

The chatter I've heard is the ai itself has figured out how to do a lot more with a lot less hardware and that the cycle may be on a cycle akin to Moore's Law but it doesn't have a name yet.

[booscrawl]

A company called PrismML figured out how to compress weights down to about 1.125 bits per weight. The theory is that they did a custom re-train of the Qwen3 8/4/1.7B models.

https://prismml.com/news/bonsai-8b

Their 8B model is about as smart as other 8B models but takes up only 1.125 bits per weight. In other words, an 8B parameter model's weights can fit in 1.125 GB of RAM, and if you need to access all the weights per token, you only need total DRAM bandwidth of 1.125 GB/s per token per second. So an 11.25 GB/s CPU could do 10 t/s, or a 400 GB/s GPU could do 355 tokens/s.

Their kernels aren't very optimized right now but their 8B model hits 133 t/s generation speed on a GPU with 800 GB/s of bandwidth. It's impressive.

Maybe in a few months we will see a 122B parameter model that fits in 17.2 GB of RAM, like Qwen3.5-122B-A10B, and only needs 1.4 GB of RAM access per token, or a 397B model that fits in 55.8 GB of RAM and only needs 2.4 GB of RAM access per token.

Edit: All the math about weight size and bandwidth ignores the fact that you have to access the KV cache constantly. Still, shrinking weights by almost 8x is a huge improvement.