How does AI DO it? Diagnosing electronic devices.

[KE5FX]

We'll see. But I think you ought to understand why a lot of the greater public, who may not have a very good idea of what the technology is or how it works, have their pitchforks out. And not just against data centers, but against the whole damn technology in general. Because in many ways it's being shoved down our throats, not so much by techies such as yourselves but by our greedy overlords who are walking around with dollar signs in their eyes.  Maybe ought to temper your enthusiasm for it a little, at least for the time being.

In my thinking, I try to separate the technology's merits from its social effects, and I generally succeed at that.  You don't want me involved in governing the latter, because I'm a full-speed-ahead kind of guy.  But I agree that it's entirely reasonable to worry about how these resources are built, accessed, and paid for.  There are paradoxes ahead with no easy resolution.

For instance, I am a big fan of local models as I said, but that's my selfishness showing through.  Due to the way these things work, running current-generation language models locally is astonishingly, criminally inefficient.  They badly need parallelism.  An inference engine has to plow through billions of weights for every token that comes out, so in any given installation, you really want to get the most work out of every weight while it's loaded into the execution unit(s).  If you have a thousand prompts running at once, you can process tokens from all active users before fetching the next weight, but individual models running on individual PCs don't benefit from that scale. 

So if we don't build those data centers, or if their computing resources are not made available to everyone who can benefit from them at a scale usually associated with government-sized organizations, the wasted power that everybody talks about will be orders of magnitude greater.  It would be better for everyone if we didn't have to mess with running our own models at home or at work, but humans no longer even pretend to behave cooperatively unless they belong to one cult or another.

This is a reason to pay attention to those who are trying to set the standards, capture the users and write the regulations.  It is not a reason to plug your ears with your fingers, rock back and forth, and mutter about how the "stochastic parrots" are "not really reasoning."  That is pure, unadulterated ignorance.  Whatever problems you anticipate with AI, whatever fears you have about it, ignorance is not going to help.

[brucehoult]

For instance, I am a big fan of local models as I said, but that's my selfishness showing through.  Due to the way these things work, running current-generation language models locally is astonishingly, criminally inefficient.  They badly need parallelism.  An inference engine has to plow through billions of weights for every token that comes out, so in any given installation, you really want to get the most work out of every weight while it's loaded into the execution unit(s).  If you have a thousand prompts running at once, you can process tokens from all active users before fetching the next weight, but individual models running on individual PCs don't benefit from that scale. 

So if we don't build those data centers, or if their computing resources are not made available to everyone who can benefit from them at a scale usually associated with government-sized organizations, the wasted power that everybody talks about will be orders of magnitude greater.  It would be better for everyone if we didn't have to mess with running our own models at home or at work,

I accept the theory, but I'm not sure the numbers add up.

My little SBC with the above examples is doing around 5 tok/s on 14W, so a million tokens will use 0.77kWh, which at domestic electricity prices here in NZ is around NZ$0.30 or US$0.18, perhaps half of that for industrial users. Or for that matter also half that if I run it between 11 PM and 7 AM (and only 20% more if I store night power in a battery to use during the day, as I actually do).

The cost of using a million tokens on OpenAI GPT5 is $10-$15, Claude $15-$25, Grok $6.

Those are all 30x to 100x more than it costs me at home.

Certainly everyone also has to pay to purchase the hardware. My box cost around $300 in early May. That's 30 months of an entry-level $10/month Claude account. Or three months of the $100 account that people seem to have to pretty quickly move to.

You can get more and better quality work out of a $10/month Claude account than a $300 local machine. But are Anthropic making money on that? I would think almost certainly not. None of those companies are making money at all. Goodness knows how much they've spent on hardware per user. Crazy.

People are doing much more serious local work on Mac Studios with 512GB RAM. Until that option was discontinued in early March upgrading a base 36GB Mac Studio to 512GB RAM was a $4000 option on top of the base $2499 price (with the minimum 512GB SSD).

If you run the maximum sized model that can fit in a 256GB or 512GB RAM Mac Studio then you are again down into the 5-10 tok/s range.

There are also privacy and security implications of local processing.

[KE5FX]

The numbers start to add up when you consume more like a million tokens per day from frontier models, and can't/don't want to wait for them.   I've invested the equivalent of a fairly decent car in hardware to run models like GLM 5.3 locally, and while I sleep better at night knowing I can do that, I can also see how poorly it's going to scale, just for my own usage alone.

Agreed, the new Macs may turn out to be pretty interesting.  Waiting for those was one option I considered.

[brucehoult]

Agreed, the new Macs may turn out to be pretty interesting.  Waiting for those was one option I considered.

Also note that standard llama-server can process multiple prompts using the same weights fetch, just like the big boys. You just pass -np 4 or whatever. In fact 4 is the default number of "slots" I get.

Internet says a Mac Studio crosses from being RAM bandwidth limited to processing limited at around OLLAMA_NUM_PARALLEL = 16 to 32 for very small small models (e.g. under 15B) and never for larger ones. For 30B to 70B you would get better overall performance at up to 4 or 8 parallel tasks.

The problem of course is that you then need to use 4 or 8 or 16 or 32 times as much of your RAM for the context for each conversation.

[Someone]

At the moment it isn't something I can use in my current role due to security concerns.

Air gapped hardware running from read only media is viable for local inference with open models. What sort of security exceeds that?

[Zero999]

Could you give a model which has only been trained up to the typical level of education of an above average 18 year old, give it a book on C programming and expect it to solve any real world problems? If so, then I would be genuinely impressed.

With that specific constraint probably not. But neither will any above average 18 year old when you give them the task to first read and memorize a C book without any access to answer any questions that might arise and no way to actually write any code. And then later ask them to solve a real world problem with only the knowledge they memorized from that book. Even open book will likely not help, since if it is not in there, it will be very hard to come up with anything reasonable, especially if time is constrained.

As for the 10 programmers, a state of the art AI model has been trained on orders of magnitude more data, but it also "knows" orders of magnitude more. Which one is greater, is likely hard to estimate.

I don't expect the student to be able to memorise the book, just keep it to hand as a reference.

To put it into perspective. When I was 16, I managed to write a simple Connect 4 program in MS QBasic. My training material consisted only of the in program help and what I had l learned at school which didn't study programming. Granted, my program wasn't the strongest player, but was better than me and did take some practice to beat it. I'm not boasting. I'm sure plenty of other people here have done far greater things when bored over the school holidays.

Up until about 5 years ago, all code had been written by humans. The first computer programs were written by those who hadn't seen any code before i.e. no training on code, just knowing what each machine code instruction did.

My point is that we're much further away from AI being able to do everything, than those who are most optimistic about it think.

I will say I'm glad I'm not in the position of someone who's about to graduate from high school or enter the workforce.  Things are going to get hairy.

The job market for junior engineers is rough.  Although, it's never been that great in the last decade or two, to be honest.

This kind of thing reflects a failure of society to properly invest in engineering.  Graduates are left to fend for themselves, and some will make it to senior roles. Companies are unwilling to invest in junior roles, because frequently juniors end up leaving companies before they have "paid back" all of the training that was implicitly spent on them.  But without juniors, where do the seniors come from?

Tough problem.  And sure, AI models make it a bit worse, since a lot of the common junior tasks can be automated by AI now, like writing test harnesses.

Apprenticeships are the answer.

Then there's the fact that engineering isn't as well paid as it used to be and quite likely never will. Perhaps technician roles are more suitable for many? Then there are jobs at smaller companies which incorporate both some tech and engineering work. In future they probably will involve using AI, but that's not a bad thing.

At the moment it isn't something I can use in my current role due to security concerns.

Air gapped hardware running from read only media is viable for local inference with open models. What sort of security exceeds that?

I'm not going to reveal the security classification because even doing that is probably not allowed.

Yes, local models are the only way. It is something the company is actively working on, but I am not involved with it.

[Siwastaja]

My little SBC with the above examples is doing around 5 tok/s on 14W

That demonstrates the problem exactly. Some Fable-like thing in agentic loop in cloud does maybe 200-300 tok/s per customer on 500W per customer (very rough guesstimates). That's better tokens/joule efficiency than yours, and it's nearly a 100x larger model (so apples-to-oranges comparison in its favor).

This is because RAM bandwidth is really expensive, also energy-wise. HBM really optimizes the RAM interface energy use compared to your DDR5(?) setup giving an order of magnitude energy improvement, and batching gives another order of magnitude. The problem is, you can't afford the hardware (HBM; hardware designed for AI inference), and even if you would, your workload wouldn't likely batch (unlike it's parallel by nature). The weights-in-mask-ROM story (TAALAS) is claiming to shift this balance.

The J/token is only relatively "good" because the model is small, so it's apples-to-oranges.

So KE5FX is right. Playing around with local models is still kind of a luxury / specialized thing; it would be impossible to change current AI use patterns (those emerged within just last 12 months!) in masses into local models simply from energy efficiency view alone. Individuals can do so. This isn't set in stone, of course, in AI everything changes at a huge speed. Expect some things that do not. I don't except RAM to become cheap again anytime soon.

But yes, it's interesting how good and consistent reasoning even small models can do nowadays. Your car wash reasoning stage is interesting read because it demonstrates how well even simple models "explore ideas" and do not fixate only getting the direct answer. This is crucial to the "intelligence" part. The question is, does the model get confused with its own exploration when the context grows.

[KE5FX]

Up until about 5 years ago, all code had been written by humans. The first computer programs were written by those who hadn't seen any code before i.e. no training on code, just knowing what each machine code instruction did

An open question is how to measure progress in this very direction.  One attempt, known as ARC-AGI, has been targeted directly at Moravec's paradox (as fallacious as it is), deliberately trying to come up with tasks that are supposedly easy for humans but hard for AI.  These are spatial-reasoning benchmarks in which the test-taker has to infer the rules from a very limited number of examples, then apply them to solve similar but previously-unseen problems. 

ARC-AGI benchmarks are now in their third generation.  The first two went from all-but-impossible for AI to completely-solved in less than a year's time.  If I recall correctly, GPT-6 Astra is doing about 60% on the ARC-AGI 3 series without a specialized test harness, and close to 100% with a harness.  I tried one or two of them myself and let's just say my scores are closer to 0%, Moravec notwithstanding.

By the same token (groan), in-context learning is more than good enough to use novel human and computer languages effectively, and has been for a while.  The 1M+ context sizes in today's LLMs is large enough to hold textbooks.  So if a model had never seen C, for example, it could read K&R on the fly and write programs in it without any additional pretraining.

So, I don't agree that "we're much further away from AI being able to do everything, than those who are most optimistic about it think."  The truth is, problems you remember being showstoppers from last year are no longer obstacles at all.  If you don't spend some time every week (if not every day) reading up on the latest results you will not be aware of this.   

And if you go months or years without updating your priors on what AI can do, as some other people in the thread seem proud of doing, your knowledge rapidly becomes worse than worthless.
 
Again, this is what I was promised as a kid, reading Tom Swift and watching Star Trek, so I'm not complaining.  Better late than never.  But it's still exhausting.  I will need to spend the entire weekend coming up to speed with the new GPT-6 release, trying to get some good work out of it before they panic and pull the plug on Monday like they did when Fable came out. 

[Siwastaja]

By the same token (groan), in-context learning is more than good enough to use novel human and computer languages effectively, and has been for a while.  The 1M+ context sizes in today's LLMs is large enough to hold textbooks.  So if a model had never seen C, for example, it could read K&R on the fly and write programs in it without any additional pretraining.

Perfect example of this is, you can ask AI to solve some super esoteric committee-driven public-money protocol or project definition which is as (unnecessarily) complicated as the whole C language for example, it reads those 20 PDFs 50-200 pages each you throw at it and writes a working implementation in 30 minutes.

Or, I asked it to run our prequalification test datasets on a custom Windows software it definitely has never seen in its training data (I have hard time believing it would have been trained with the user manual of that software; it is remotely possible but unlikely), and there it goes, takes screencaps of it and commands mouse clicks and figures out what every setting does. That's in-context creativity.

The misconception is that training is needed to be able to analyze or act on information. No, that information in context is fine. But training is needed so that it understands what it does not understand so that it can fetch the right documents for example. It's the same as with a human, you can take a seasoned all-variant expert who knows a lot about everything and throw them nearly any task, and they know enough of everything they can find the right books, do the right Google searches, do the right experiments (e.g., write test programs and see what happens), talk with the right people, ask the right questions, and get nearly anything done.. But you can't give all those tasks to 10-year old smart kid even if their cognitive capabilities are already on the same level - even if they are able to research, they have so much to learn it takes a lot of time and fills their context window (working memory) - even if they are efficient at it.

With AI, context window is already pretty large, but if it needs to learn everything, 1M tokens will run out quickly (also running 100 turns creating 200k context just to figure something out in a "small" model is still a whole lot of wasted memory BW!). Hence, we have models trained on how to use grep, with all SQL commands, how aws command line tool work and what kind of machines Amazon offers, the C standard, the Python tutorials, how thermodynamics and biology works and so on. So that once you ask it to do something in your imaginary programming language, it can do it exactly because it has the context free for that because it doesn't need to learn everything else at the same time. Good training and model size = beneficial, but not absolute requirement at all for creative problem solving. Knowledge frees up creativity. Just like in human brain. Poorly educated can make large discoveries, but rarely do so because they spend more time figuring out the basics. Even if they are equally smart.

[Randy222]

In my thinking, I try to separate the technology's merits from its social effects, and I generally succeed at that. 

What was your outcome for the microwave, TV dinner, cell phone, internet, facebook?

[KE5FX]

In my thinking, I try to separate the technology's merits from its social effects, and I generally succeed at that. 

What was your outcome for the microwave, TV dinner, cell phone, internet, facebook?

Generally positive, I guess.  I'm waiting on my TV dinner to finish cooking in the microwave now, while using the Internet.  Cell phones are certainly a boon unless you feel that it should be more difficult and expensive for people to keep in touch with each other.

As for Facebook, well... intentionally capturing, controlling, and corrupting human interaction for profit is hard to defend, so I wouldn't try.  Somebody else'll have to speak up for them, ideally in some other thread.

[brucehoult]

My little SBC with the above examples is doing around 5 tok/s on 14W

That demonstrates the problem exactly. Some Fable-like thing in agentic loop in cloud does maybe 200-300 tok/s per customer on 500W per customer (very rough guesstimates). That's better tokens/joule efficiency than yours, and it's nearly a 100x larger model (so apples-to-oranges comparison in its favor).

I don't think you have a source for that Watts or tokens/Joule comparison.

Also, there is nothing magical about the hardware used by Anthropic. Any company could buy the same hardware and run it with only a few months worse trillion-parameter open weights model in-house.

Also there is a torrent of cheap high performance AI hardware coming. Here is just one of them.

https://medium.com/@arnis.us/400-tokens-per-second-on-a-12-000-tenstorrent-quietbox-425aaf55bbeb

[Siwastaja]

I don't think you have a source for that Watts or tokens/Joule comparison.

Source for watts is that I, of course, asked Claude to estimate and it based it on actual hardware usually used in datacenters divided by some assumed usual number of parallel customers running on that service, and came with a 200-1000W range for a tight agentic loop. Tokens per second is own observation.

These ranges pass all sanity checks. I don't buy the story that I would be using tens of kilowatts when I ask Claude to do something. Tens of kilowatts is what the whole cluster uses when it serves many tens of customers. 500W sounds like a very sensible number.

In any case, inference energy is not really the issue per user. It's maybe 0.1% of Anthropic's pay-per-use API fees (which are huge) and if you max out your max plan then it might be 5-10%. At home, energy also isn't the issue, even if it's less efficient. It's still not a problem. Energy only becomes a problem at a scale - if humanity does a lot of unnecessary or harmful work with AI, then the modest energy usage per user piles up.

Also, there is nothing magical about the hardware used by Anthropic. Any company could buy the same hardware and run it with only a few months worse trillion-parameter open weights model in-house.

Of course. Anthropic buys processing from same vendors everyone else, and they use hardware anyone can buy (with deep enough pockets). This is HBM stuff, anyone can build a similar cluster for maybe $100k-$200k. For a company it's not a big investment after all. Companies regularly can spend more than that on a lab (just the room alone, furniture, real estate cost, not even to talk about expensive test gear) that is used 1% of the time.

But a regular home user can't afford HBM because it's many times more expensive than DDR5 which home users can't afford either currently, and regular home users can't easily batch their processes. These are the facts that are impossible to circumvent right now. Things might be very different in 2-3 years though. Maybe even sooner.

So currently, home users have worse energy efficiency and compensate by using much smaller and simpler models. And it's interesting how capable they still are at many types of tasks.