Memory management bug in Intel CPUs threatens massive performance hits.

[Richard Crowley]

This is what worries me most: "can be potentially exploited in JavaScript running in a web browser". Right now, here, as we type...

The rest of the paragraph reads....

...Apple will release an update for Safari on macOS and iOS in the coming days to mitigate these exploit techniques. Our current testing indicates that the upcoming Safari mitigations will have no measurable impact on the Speedometer and ARES-6 tests and an impact of less than 2.5% on the JetStream benchmark. We continue to develop and test further mitigations within the operating system for the Spectre techniques, and will release them in upcoming updates of iOS, macOS, and tvOS. watchOS is unaffected by Spectre.
https://support.apple.com/en-us/HT208394

[bd139]

I think its not disputed that NSA attempts to get hardware manufacturers to include back doors in hardware. What would be surprising is if there were not any backdoors, not if there were.

I wouldn't count on any modern CPU, firmware (or perhaps even OS) being free of these kinds of issues because they may be a feature, not a bug.

ya know..

Shoo!

And heartbleed was not an accident...

Heartbleed was definitely an accident. Ive written a few things before with exactly the same cock up in it.

Two entire people were responsible for maintain OpenSSL which is the foundation of a big chunk of all public facing crypto on the planet. You can’t expect even the best two people not to miss some fuck ups in a piece of software written in one of the least well defined languages of all time (C).

This is what worries me most: "can be potentially exploited in JavaScript running in a web browser". Right now, here, as we type...

The rest of the paragraph reads....

...Apple will release an update for Safari on macOS and iOS in the coming days to mitigate these exploit techniques. Our current testing indicates that the upcoming Safari mitigations will have no measurable impact on the Speedometer and ARES-6 tests and an impact of less than 2.5% on the JetStream benchmark. We continue to develop and test further mitigations within the operating system for the Spectre techniques, and will release them in upcoming updates of iOS, macOS, and tvOS. watchOS is unaffected by Spectre.
https://support.apple.com/en-us/HT208394

This is because the timers in JS have enough resolution to be able to reduce cache read times. They are merely removing timer resolution. Firefox has already done this as of v57.

[bd139]

On the matter of virtualisation, it’s a hack in x86. Also:

https://marc.info/?l=openbsd-misc&m=119318909016582

[RoGeorge]

Still don't get it.
....
How can I read those results before being discarded?

The computational result isn't stored, but the trace of it having been...

Just finished reading the original paper for Meltdown, https://meltdownattack.com/meltdown.pdf. The video does its best, but it was not enough, and yes, the vulnerability is as bad as it can be.

The attack is very clever indeed, but I found the paper totally worth reading it not only for describing the attack, but especially for describing the principles about speculative execution and out of order execution in general, and Intel implementation in special.

[Corporate666]

Yes indeed. It doesn’t look good for the IT business at all. I have, as someone deeply involved in the security side of things, considered cashing everything I have in and bailing. It’s too bloody stressful keeping the snowflakes covered in piss alive (google “programming sucks” for context of that comment).

Sorry for derailing the thread topic a bit, but as (primarily) a business guy myself who started out as a technical guy (software, the mechanical engineering, then electronics) - the above comment sounds like music to my ears, from a business standpoint.   In other words, you are an expert in a field that is full of fast-paced change, commotion and where there are always new emergencies and endless numbers of "snowflake" clients who need fires put out and assurances given.  That sounds like a recipe for high income, being able to be picky about who you take on as clients, and essentially shooting fish in a barrel.  What makes you want to cash out your chips and get out?  The stress? 

[JoeN]

How exactly are the speculative results extracted?
How come that the speculated values can still leave side effects behind, even after discarding the results?
What are those side effects, and how are they used to access a miss predicted and discarded calculation?

If I understood the video correctly, the exploits take advantage of the {timing] information whether or not some [injected] value has been cached by the CPU or not, due to the speculative nature of execution of the instructions of the modern CPUs. You just need to make the CPU to fetch some known data from the memory and use the available high resolution on-chip timers to measure how long does it take to execute that data fetch. If the execution time is "fast", the value was cached and if the execution time was "slow" the value was not in the cache. By using this direct timing information one can extract indirectly the wanted information for the exploit.

The analogy I am using for non-technical people is the CPU basically has a gambler's "tell".  The gambler won't tell you his card, each time you ask him if it is a deuce or a three or a four or what he says "piss off".  But unfortunately for him, he says it a lot faster when you actually asked the right question.

[raptor1956]

So, what are the odds that the NSA and GCHQ and many other government signals intelligence operations were unaware of this?  Wanna bet these exploits are in current use by some of the above?


Brian

[mtdoc]

The ironic thing is that while this is may cause a brief hit to Intel's rep, in the end it probably means selling a whole bunch of new chips. 

And new computer sales for Apple, HP, Dell,  etc, etc. which means new sales for storage, memory and other peripheral makers...

Based on the stocks reaction today, I think the market may be coming to realize this.

Maybe they can just make all computers disposable with a 1 year shelf life - that'll  keep the tech market pumping...

[bd139]

Yes indeed. It doesn’t look good for the IT business at all. I have, as someone deeply involved in the security side of things, considered cashing everything I have in and bailing. It’s too bloody stressful keeping the snowflakes covered in piss alive (google “programming sucks” for context of that comment).

Sorry for derailing the thread topic a bit, but as (primarily) a business guy myself who started out as a technical guy (software, the mechanical engineering, then electronics) - the above comment sounds like music to my ears, from a business standpoint.   In other words, you are an expert in a field that is full of fast-paced change, commotion and where there are always new emergencies and endless numbers of "snowflake" clients who need fires put out and assurances given.  That sounds like a recipe for high income, being able to be picky about who you take on as clients, and essentially shooting fish in a barrel.  What makes you want to cash out your chips and get out?  The stress? 

You’re right about the recipe. I am however entirely immune to stress. I’m the sort of person who sits there leisurely eating a Cornish pasty while the world burns around me. You don’t solve any problems by getting stressed. Occasionally smashing something that has smited you is recommended however (hat tip to Mr Widlar  for that one)

The problem is my brain. I can see the whole abstraction of the machine in my mind, vast networks spanning thousands of nodes and zoom in and out right down to individual components and even lines of code. I can feel it breathing, see where it is sick, see data flows and bottlenecks instantly. I’m sure any programmer understands the moment this clicks (and then the moment someone taps on your shoulder and it all goes away in a puff of smoke).

Problem is none of this really exists and is changing so fast and this screws your mind up over time. Unlike a JVM, you don’t have a garbage collector up there. Makes you sick. Sometimes I just phase out unable to switch between the two worlds.  It requires so much space that it pushes things that are important out. My wife can recall so many things going back 20 years. I can’t. Even some memories of my children are vague when they were very young. I attribute this to information overload. Now I can remember which methods to call on windows workflow foundation SQL persistence engine to get it to dance like the monkey it is but this is of no value now as the information is transient as I haven’t used it for nearly 8 years.

Some people attribute this to burn out but it’s something different and far more worrying. I know a few people who have bailed already on this basis. One guy even went mental and shit on his bosses chair and threw himself under a bus, which uneventfully stopped before it ran him over and the driver called an ambulance. Most people I work with are addicts of some kind also.

Ergo I suppose I worry about a cross of mental health and the value of the information I am processing over time. It’s not good for you.

Therefore I’m  taking the cash I need out as quickly as possible and filling what precious time and headspace I have with things I care about.

And there you have it.

[Corporate666]

Yes indeed. It doesn’t look good for the IT business at all. I have, as someone deeply involved in the security side of things, considered cashing everything I have in and bailing. It’s too bloody stressful keeping the snowflakes covered in piss alive (google “programming sucks” for context of that comment).

Sorry for derailing the thread topic a bit, but as (primarily) a business guy myself who started out as a technical guy (software, the mechanical engineering, then electronics) - the above comment sounds like music to my ears, from a business standpoint.   In other words, you are an expert in a field that is full of fast-paced change, commotion and where there are always new emergencies and endless numbers of "snowflake" clients who need fires put out and assurances given.  That sounds like a recipe for high income, being able to be picky about who you take on as clients, and essentially shooting fish in a barrel.  What makes you want to cash out your chips and get out?  The stress? 

You’re right about the recipe. I am however entirely immune to stress. I’m the sort of person who sits there leisurely eating a Cornish pasty while the world burns around me. You don’t solve any problems by getting stressed. Occasionally smashing something that has smited you is recommended however (hat tip to Mr Widlar  for that one)

The problem is my brain. I can see the whole abstraction of the machine in my mind, vast networks spanning thousands of nodes and zoom in and out right down to individual components and even lines of code. I can feel it breathing, see where it is sick, see data flows and bottlenecks instantly. I’m sure any programmer understands the moment this clicks (and then the moment someone taps on your shoulder and it all goes away in a puff of smoke).

Problem is none of this really exists and is changing so fast and this screws your mind up over time. Unlike a JVM, you don’t have a garbage collector up there. Makes you sick. Sometimes I just phase out unable to switch between the two worlds.  It requires so much space that it pushes things that are important out. My wife can recall so many things going back 20 years. I can’t. Even some memories of my children are vague when they were very young. I attribute this to information overload. Now I can remember which methods to call on windows workflow foundation SQL persistence engine to get it to dance like the monkey it is but this is of no value now as the information is transient as I haven’t used it for nearly 8 years.

Some people attribute this to burn out but it’s something different and far more worrying. I know a few people who have bailed already on this basis. One guy even went mental and shit on his bosses chair and threw himself under a bus, which uneventfully stopped before it ran him over and the driver called an ambulance. Most people I work with are addicts of some kind also.

Ergo I suppose I worry about a cross of mental health and the value of the information I am processing over time. It’s not good for you.

Therefore I’m  taking the cash I need out as quickly as possible and filling what precious time and headspace I have with things I care about.

And there you have it.

Understood completely.  I have exactly the same situation.   I have a few different technical areas I work in at my job which are quite separate and different and require a lot of time to keep technically proficient in.  I really enjoy each of these different fields, but juggling all of that plus running a business and all the associated tasks including manufacturing/production means I always have a million things going on.  I joke that I have the memory of a goldfish - I forget everything that happened more than 2 minutes ago.  I use precisely the same description as you - that so much data goes into my brain constantly that most stuff gets squeezed out, leaving me forgetting many things most other people remember.

I don't have a wife/kids but I can definitely see how others would feel you are not "present" enough with them when you suffer from such information overload that you don't recall things and they feel it indicates a lack of care.  It doesn't, I know, but I am sure it can appear that way to others.

Well, cheers to you mate for recognizing it as a potential issue and addressing it.  Takes a solid husband and father to do so.  My respect.

[floobydust]

Linus Torvald called it out:

...
> Any speculative indirect calls in the kernel can be tricked
> to execute any kernel code, which may allow side channel
> attacks that can leak arbitrary kernel data.

"Why is this all done without any configuration options?

A *competent* CPU engineer would fix this by making sure speculation doesn't happen across protection domains. Maybe even a L1 I$ that is keyed by CPL.

I think somebody inside of Intel needs to really take a long hard look at their CPU's, and actually admit that they have issues instead of writing PR blurbs that say that everything works as designed.

.. and that really means that all these mitigation patches should be written with "not all CPU's are crap" in mind.

Or is Intel basically saying "we are committed to selling you shit forever and ever, and never fixing anything"?

Because if that's the case, maybe we should start looking towards the ARM64 people more.

Please talk to management. Because I really see exactly two possibilities:

 - Intel never intends to fix anything

OR

 - these workarounds should have a way to disable them.

Which of the two is it?"

                   Linus

https://lkml.org/lkml/2018/1/3/797

[nctnico]

Some people attribute this to burn out but it’s something different and far more worrying. I know a few people who have bailed already on this basis. One guy even went mental and shit on his bosses chair and threw himself under a bus, which uneventfully stopped before it ran him over and the driver called an ambulance. Most people I work with are addicts of some kind also.

Ergo I suppose I worry about a cross of mental health and the value of the information I am processing over time. It’s not good for you.

What helps is to take up a hobby which doesn't need much thinking but keeps you busy. I'm not a sports person at all but I took up swimming a couple of years ago and it helps to clear&organise my mind.

[Mr. Scram]

I think its not disputed that NSA attempts to get hardware manufacturers to include back doors in hardware. What would be surprising is if there were not any backdoors, not if there were.

Nobody's disputing this, but this thread is not about that. Neither are all the other threads you insist on making into conspiracy stories. My remark was about the continuous pushing of your agenda and derailing of threads.

Don't get me wrong, you seem like a nice guy I could have a drink with, but the persistence is tiring.

[Mr. Scram]

Anyone thinking that all of this seem a bit complicated and weird, it should be pointed out that nowadays, the encryption on a laptop (some of it I guess) can be broken from recording and analyzing the noise patterns coming from the laptop when measuring the sound with a recording device close by. Pretty obscure stuff.

Do you have a link? I think that's the kind of side channel attack that gets a fair bit of attention in regards to mitigation. Of course, an infected laptop could send out intentional sounds or signals that can be used to break encryption. That's a given, but you need to have a foothold already and in those cases you generally have more effective methods to extract data.

[bd139]

Some people attribute this to burn out but it’s something different and far more worrying. I know a few people who have bailed already on this basis. One guy even went mental and shit on his bosses chair and threw himself under a bus, which uneventfully stopped before it ran him over and the driver called an ambulance. Most people I work with are addicts of some kind also.

Ergo I suppose I worry about a cross of mental health and the value of the information I am processing over time. It’s not good for you.

What helps is to take up a hobby which doesn't need much thinking but keeps you busy. I'm not a sports person at all but I took up swimming a couple of years ago and it helps to clear&organise my mind.

Agree entirely. Exercise is a winner every time as well. I’m not a sports person but I found I really like running. Unfortunately this makes me hungry so I ran about 7 miles earlier this week then went in KFC on the way back and consumed my body weight in chicken 

Anyone thinking that all of this seem a bit complicated and weird, it should be pointed out that nowadays, the encryption on a laptop (some of it I guess) can be broken from recording and analyzing the noise patterns coming from the laptop when measuring the sound with a recording device close by. Pretty obscure stuff.

Do you have a link? I think that's the kind of side channel attack that gets a fair bit of attention in regards to mitigation. Of course, an infected laptop could send out intentional sounds or signals that can be used to break encryption. That's a given, but you need to have a foothold already and in those cases you generally have more effective methods to extract data.

Rubber hose cryptography is better in this situation

Then again this problem predates computers. My wife’s grandfather was the designer of “quiet rooms” used by the British government around the Cold War era. They, even in the 1960s has worked out you could listen in on conversations by listening to the sounds transmitted through heating pipes in and out of the rooms. They even had rudimentary expertise on deciphering chunks of documents that were being typed from recordings by “golden eared” experts of the pipe sounds.

[Mr. Scram]

The ironic thing is that while this is may cause a brief hit to Intel's rep, in the end it probably means selling a whole bunch of new chips. 

And new computer sales for Apple, HP, Dell,  etc, etc. which means new sales for storage, memory and other peripheral makers...

Based on the stocks reaction today, I think the market may be coming to realize this.

Maybe they can just make all computers disposable with a 1 year shelf life - that'll  keep the tech market pumping...

Unlike many companies think, the world isn't made of money. You can't keep buying new kit and you can't keep migrating. The pace is already quite taxing as it as and adding to it might break the camel's back. There's some room, but buying new computers the whole world over simply isn't an option. Any organisation bigger than tiny is constantly renewing itself to maintain the status quo, stuffing bricks back in the crumbling wall. Many organisations are sitting ducks in a world where cybercrime is rapidly becoming one of the largest and most profitable businesses.

Maybe even more importantly, there's no guarantee the next one won't pop up next month. We've had various hardware dependent attacks the past year. You can't keep buying new stuff every time, not even having it fully deployed when the next one hits.

[mtdoc]

I think its not disputed that NSA attempts to get hardware manufacturers to include back doors in hardware. What would be surprising is if there were not any backdoors, not if there were.

Nobody's disputing this, but this thread is not about that. .

No, I think it is relevant. What are the odds that the NSA was not aware of this and already exploiting it?

Was this really an unintentional "bug"

[Mr. Scram]

No, I think it is relevant. What are the odds that the NSA was not aware of this and already exploiting it?

Was this really an unintentional "bug"

What's the use speculating about that? We won't know, until someone releases the documents. We know that they look for these things, even try to plant them, but we don't know if that's the case here. We do know that we tend to attribute to malice what is actually stupidity. Maybe it is, maybe it's not. We can argue yes or no all we want, but we won't get closer to the truth.

And again, it's also about making every single thread into a conspiracy. It's tiring.

[bd139]

The ironic thing is that while this is may cause a brief hit to Intel's rep, in the end it probably means selling a whole bunch of new chips. 

And new computer sales for Apple, HP, Dell,  etc, etc. which means new sales for storage, memory and other peripheral makers...

Based on the stocks reaction today, I think the market may be coming to realize this.

Maybe they can just make all computers disposable with a 1 year shelf life - that'll  keep the tech market pumping...

Unlike many companies think, the world isn't made of money. You can't keep buying new kit and you can't keep migrating. The pace is already quite taxing as it as and adding to it might break the camel's back. There's some room, but buying new computers the whole world over simply isn't an option. Any organisation bigger than tiny is constantly renewing itself to maintain the status quo, stuffing bricks back in the crumbling wall. Many organisations are sitting ducks in a world where cybercrime is rapidly becoming one of the largest and most profitable businesses.

Maybe even more importantly, there's no guarantee the next one won't pop up next month. We've had various hardware dependent attacks the past year. You can't keep buying new stuff every time, not even having it fully deployed when the next one hits.

What would be nice is FPGA fabric and self reconfigurable computers. Then you can keep a base abstraction which is formal rather than a pile of hacks. If there’s a problem, reconfigure the hardware.

This is a lower level than microcode.

[Mr. Scram]

Some of the exploits Ive read about seem to allow arbitrary code running on a VM to access code running in a higher ring which is supposed to be invisible to the OS, allowing information from other VMS or running on the hardware3 above the kernel and OS space to be accessed..

This all begs the question, what and how do people know when OS's and virtualization end?

Recently it turned out that many processors from one manufacturer had an entire separate CPU with an embedded OS, Minix, on the same die, which could access everything running in the main CPU's OS space, image the drive, etc,  including when the processor was supposed to be powered off or hibernating.  This evil twin OS ran in something called "ring -3" and it even had a web server. Some traffic going over the net also never was seen by the OS, it went straight to this other CPU.

You can read more here:

http://www.cs.vu.nl/~ast/intel

Not one manufacturer, both major x86 manufacturers. AMD calls it TrustZone and actually has an ARM processor embedded. They aren't some hidden secret either, but sold as a management tool. It's a huge boon not having to manually turn on 2500 computers to run an update. What's new is that the theoretical risks have seen for years have now been converted into an actual, practical thread by a vulnerability and the system being dissected and understood ever more. The security through obscurity started cracking in major ways.

[Mr. Scram]

What would be nice is FPGA fabric and self reconfigurable computers. Then you can keep a base abstraction which is formal rather than a pile of hacks. If there’s a problem, reconfigure the hardware.

This is a lower level than microcode.

I've thought about this the past year a lot. Doing it that way solves a number of problems, but creates others. Having changeable hardware under the hood means never knowing what is actually going on. You'd need some independent way of verifying the configuration of the chip and if it's hardware doing that you're back to square one, if it's a configurable fabric it's turtles all the way down.

[cdev]

But, here's the problem, the author of Minix, who is kind of a luminary in the world of computing, and a significant number of other people far more knowledgeable than myself or I venture to say likely yourself as well, were disturbed enough about this to make a stink about it. And the description I have read about it it doesn't look like that is all it is to me.

Even if that was the intent, then shouldn't it not be present on cheaper HW, since that HW is basically meant to be used by consumers, not in servers, and is basically disposable?  But, it is.

Were it just a routine system administration tool, for which the internals were known and public, as you portray it as, the outcry - which was focused on security and privacy implications, would not have happened.

Some of the exploits Ive read about seem to allow arbitrary code running on a VM to access code running in a higher ring which is supposed to be invisible to the OS, allowing information from other VMS or running on the hardware3 above the kernel and OS space to be accessed..

This all begs the question, what and how do people know when OS's and virtualization end?

Recently it turned out that many processors from one manufacturer had an entire separate CPU with an embedded OS, Minix, on the same die, which could access everything running in the main CPU's OS space, image the drive, etc,  including when the processor was supposed to be powered off or hibernating.  This evil twin OS ran in something called "ring -3" and it even had a web server. Some traffic going over the net also never was seen by the OS, it went straight to this other CPU.

You can read more here:

http://www.cs.vu.nl/~ast/intel

Not one manufacturer, both major x86 manufacturers. AMD calls it TrustZone and actually has an ARM processor embedded. They aren't some hidden secret either, but sold as a management tool. It's a huge boon not having to manually turn on 2500 computers to run an update. What's new is that the theoretical risks have seen for years have now been converted into an actual, practical thread by a vulnerability and the system being dissected and understood ever more. The security through obscurity started cracking in major ways.

[stj]

they arent called "INTEL" for nothing!!
hell, they arent even designed in the west - think about that for a second!!!
i'm pretty sure that breaks rules relating to military procurement. 

[Mr. Scram]

But, here's the problem, the author of Minix, who is kind of a luminary in the world of computing, and a significant number of other people far more knowledgeable than myself or I venture to say likely yourself as well, were disturbed enough about this to make a stink about it. And the description I have read about it it doesn't look like that is all it is to me.

Even if that was the intent, then shouldn't it not be present on cheaper HW, since that HW is basically meant to be used by consumers, not in servers, and is basically disposable?  But, it is.

Were it just a routine system administration tool, for which the internals were known and public, as you portray it as, the outcry - which was focused on security and privacy implications, would not have happened.

I can't put this any more gently than that it seems that you're filling the gaps of your knowledge with your imagination. The tool being present was well know. It has been a black box for quite a while, but criticized because of exactly that too. I have included a link to the FAQ of the open source BIOS Libreboot, which doesn't support processors with Intel ME. It explains in some detail what it is, does and what its capabilities are. It also includes links to other independent pages with similar information. The page dates July 2015, and isn't manipulated after the fact as I read it myself around that time. There are many other sources with similar information which pre-date this page significantly. One of the links is for instance dated June 2014.

The recent uproar was because it became clear the black box was showing cracks. The thing hidden from sight could now be seen by many people and the protection the obscurity was supposed to bring was gone. Despite the Intel ME and it's capabilities being known, its exact inner workings weren't known. One of the things discovered was that it actually runs MINIX, much to the surprise of the author of that software.

The Intel ME may have been a surprise to the general public, but it hardly was a secret. People who know what they're talking about have been fearing what would inevitably happen for years and the actual source of the uproar was that it was the big "told you so" moment everyone knew was coming. The exposure merely meant the public at large finally caught wind of it.

So please, keep this thread clear of the speculations and theories you tend to line other threads with. The subject is complicated enough as it is and many people already have trouble understanding what actually is going on without FUD being mixed in.

https://web.archive.org/web/20150730233729/http://libreboot.org:80/faq/#intelme
https://web.archive.org/web/20150908031804/https://www.fsf.org/blogs/community/active-management-technology

[Cerebus]

No, I think it is relevant. What are the odds that the NSA was not aware of this and already exploiting it?

Was this really an unintentional "bug"

No, it wasn't, on the balance of probabilities, deliberate.

I can quite see how the engineers would miss this. Their targets would have been meeting performance goals and providing the security specified by the architecture, not meeting the security goals that someone with adversarial security experience would consider desirable -  which would include quashing any possible side channels. (I can tell you from experience of trying to design systems to be covert channel free that this is very hard to do on small systems, and immensely hard to do on large complex systems like the super-scalar, out of order, execution engines that modern CPUs are.)

Speculative execution (and super-scalar processors) are all about trying to reduce latency. Protection mechanisms introduce latency. So you try and run the protection checks in parallel with the speculative execution and only stop the speculative execution once you've got results from the protection checks. This means that you will almost certainly use some protected data for speculative execution before you know the results of the protection checks for that data If you don't, you lose some of the latency advantages of speculative execution.

However, this has side effects, one of which - as we have seen - is polluting the cache with speculative fetches. An adversarial security-minded mindset would have spotted this as an information leak and at least provided an option to stall the speculative execution pipeline with interlocks between protected actions and  protection check results, resulting in no cache pollution and hence no information leak.

The problem is one of designing the chipset with a performance mindset and not being aware of the security trade-offs of some of those performance enhancing tricks. In a performance mindset it's OK that a speculative execution that falls foul of a protection check simply fails to retire* those instructions rather than undoes all the side effects of that speculative execution. Done that way there is no explicit access to that data and the architectural security model is satisfied. As we have seen, this is not enough to satisfy an adversarial security model that is intolerant of implicit partial data leaks.

*retire in this sense means 'write back the results to architectural registers once the speculative execution becomes non-speculative'.