Test Equipment Anonymous (TEA) group therapy thread

[mnementh]

Ooh nixies. I want nixies at the moment actually. There are a few nixie encrusted things here on ebay but nothing priced anything other than insanely. 

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today 

Ugghh... when Fate takes a dump on you, she piles it on. No time for TEA?!?  And just to twist the knife, a death in the family AND a shitstorm at work.     Here, have some of this:



It's green.   

The other swiss army knife - Wenger. Automatic watch for size. The saw is excellent.

Rainer

And Wenger is the one that was actually Swiss Army issue, back when the name was coined. A shame they were assimilated by the mediocre collective. Size-wise tho, I generally prefer the feel of ~84mm scales to the larger ~91mm scales in my pocket. With a well-made pocket knife, the difference is just peen. 

mnem

[Kosmic]

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19)...

That suck 

All my condolences.

[bd139]

In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock

Threads they said. A good idea they said 

The memory is, as you mention, basically L4 cache here for the disk.

[bd139]

It's green.   

Aldebaran whiskey

[mnementh]

So, cyclin_al, Kosmix, mnem, et al, what's the story with the somewhat recent (a year or two?) surge in TE from eBay sellers in Canada? I don't recall so many before. Are there a lot of businesses/labs upgrading or something?

Not sure. The federal government has been investing a lot in R&D. That might explain the influx of TE on ebay from public and private sources.

Mmmhmmm... might be they can see the writing on the wall and are looking to insulate their infrastructure from that of the US; maybe even build up some of their own STEM resources instead of borrowing from the neighbors. I've noticed some major changes in how my IPs are routed now vs when I first got here; even in the time since I started using my VPN. Used to be everything was routed through NYC and Jersey; now about half the time it goes through Montreal and sometimes even LA.

This of course can simply be my own browsing habits, or a single bit of upgraded backbone somewhere along the way. But if my critical infrastructure were dependent upon the US not losing its collective mind, I'd certainly be looking for anywhere else in the world to be connected, at least as a backup. 

Kosmic, have you noticed any such changes, or have you always been routed primarily through Montreal since you're nor'east of me?

mnem
 

[mnementh]

It's green.   

Aldebaran whiskey

Aye... I'm all out of Saurian Brandy.   

mnem
I blame Kosmic.

[bd139]

It's green.   

Aldebaran whiskey

Aye... I'm all out of Saurian Brandy.   

mnem
I blame Kosmic.

It's good job that Romulan Ale is all I'll ever need 

[mnementh]

In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock

Threads they said. A good idea they said 

The memory is, as you mention, basically L4 cache here for the disk.

So basically too many people with authority (and not enough intelligence to use it properly) to lock portions of the db, causing resources that need to updated to be unavailable for update?

http://techgenix.com/azure-locks/

mnem
"The only limitless resource is human stupidity."

[Martin.M]

germany looks for a solution

https://www.wellenkino.de/557/viewtopic.php?f=5&t=305

 

[mnementh]

It's green.   

Aldebaran whiskey

Aye... I'm all out of Saurian Brandy.   

mnem
I blame Kosmic.

It's good job that Romulan Ale is all I'll ever need 

Made you laugh.   

mnem
You're welcome.

[Kosmic]

Kosmic, have you noticed any such changes, or have you always been routed primarily through Montreal since you're nor'east of me?

I have no idea, I never do traceroute. Might be quicker to go through Montreal since it's probably well connected with NYC.

Also depend on your provider setup. My previous provider was Teksavvy, and even though I'm in Montreal, all my traffic was routed through Toronto.

[Kosmic]

germany looks for a solution

https://www.wellenkino.de/557/viewtopic.php?f=5&t=305

 

Fit perfectly 

[worsthorse]

@bd139 sorry to hear about your father-in-law.  sounds like a perfect storm hitting.  i hope you work through the stuff you have some control over quickly and that the rest of it blows over without too much damage. 

[tggzzz]

In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock

Threads they said. A good idea they said 

The memory is, as you mention, basically L4 cache here for the disk.

I forgot simple incompetence repeating old errors; silly me.

Threads are fine, up to a limit of two per physical core Beyond that you want "job fragments" in a queue, and those are handled by one of the few worker threads. That can be extremely scalable.

[Cerebus]

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today 

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

[mnementh]

Kosmic, have you noticed any such changes, or have you always been routed primarily through Montreal since you're nor'east of me?

I have no idea, I never do traceroute. Might be quicker to go through Montreal since it's probably well connected with NYC.

Also depend on your provider setup. My previous provider was Teksavvy, and even though I'm in Montreal, all my traffic was routed through Toronto.

Hmmm... I think Montreal has independent fiber going to London/EU; was guessing .ca might be trying to reprioritize traffic. Should be interesting to watch, especially around election time.

mnem
moo.

[Cerebus]

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock

Threads they said. A good idea they said 

That's why I'm a big fan of lock-free/wait-free architectures. However, they do need the programming to be done by grown-ups who understand how to take advantage of the 'still make progress' possibilities of that architecture rather than cargo culting them and just treating them as a new kind of spin lock.

[bd139]

I forgot simple incompetence repeating old errors; silly me.

Threads are fine, up to a limit of two per physical core Beyond that you want "job fragments" in a queue, and those are handled by one of the few worker threads. That can be extremely scalable.

It's always the boring errors when solving boring problems.

That's technically what this is supposed to be doing. There should be one worker per core. However somewhere along the line idiocy crept in and the consumer end of the queue consumes the entire queue until there is no work spawning a thread each. It just didn't explode until the queue was busy enough for the entire machine's resources being pissed away by the cumulative spinlocks whch eventually exceed the ability for it to process work leading to a lock concurrency issue and deadlock. Which happened after it working fine for a couple of years by the looks. Obviously then this is a false assumption by people originally looking at it that "it worked fine up until now so I don't see what the issue is" 

The horrible outcome is actually caused by this line of code which is courtesy microsoft. https://referencesource.microsoft.com/#system.core/system/threading/ReaderWriterLockSlim/ReaderWriterLockSlim.cs,1662 ... this is used by all "thread safe" (hahahaha) data structures. This is used as a lock implementation to aggressively cache data across all the threads to speed up a poorly written processing implementation.

Looking at the problem it solves it was solved with a sledge hammer rather than some grey matter so I'm going to look at cheaper ways to solve the problem than throwing it at £50k of hardware 

At least it's a distracting problem

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock

Threads they said. A good idea they said 

That's why I'm a big fan of lock-free/wait-free architectures. However, they do need the programming to be done by grown-ups who understand how to take advantage of the 'still make progress' possibilities of that architecture rather than cargo culting them and just treating them as a new kind of spin lock.

There are a 100 lessons in your sentence there which 100 people I know have never and will never learn.

Edit: correct uncaffeinated use of words.

[mnementh]

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today 

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

No, if he were BOFH-minded he'd have known the shit-shower was coming, and offed dad-in-law (via strategic ethernet probing with HV) in advance; thereby never showing up to work at all.

mnem
If you think education is expensive, try ignorance.

[bd139]

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today 

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

I have come to the conclusion I live for the shit showers. I'm not motivated unless I am about to drown

That might explain all those Teks 

[Kosmic]

In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock

Threads they said. A good idea they said 

The memory is, as you mention, basically L4 cache here for the disk.

I forgot simple incompetence repeating old errors; silly me.

Threads are fine, up to a limit of two per physical core Beyond that you want "job fragments" in a queue, and those are handled by one of the few worker threads. That can be extremely scalable.

Or you use Fibers 

Being able to suspend application code flow is a really nice and non intrusive way to implement parallelism. You end up with 1 thread per core and tons of fibers.

[mnementh]

It's always the boring errors when solving boring problems.

That's technically what this is supposed to be doing. There should be one worker per core. However somewhere along the line idiocy crept in and the consumer end of the queue consumes the entire queue until there is no work spawning a thread each. It just didn't explode until the queue was busy enough for the entire machine's resources being pissed away by the cumulative spinlocks whch eventually exceed the ability for it to process work leading to a lock concurrency issue and deadlock. Which happened after it working fine for a couple of years by the looks. Obviously then this is a false assumption by people originally looking at it that "it worked fine up until now so I don't see what the issue is" 

The horrible outcome is actually caused by this line of code which is courtesy microsoft. https://referencesource.microsoft.com/#system.core/system/threading/ReaderWriterLockSlim/ReaderWriterLockSlim.cs,1662 ... this is used by all "thread safe" (hahahaha) data structures. This is used as a lock implementation to aggressively cache data across all the threads to speed up a poorly written processing implementation.

Looking at the problem it solves it was solved with a sledge hammer rather than some grey matter so I'm going to look at cheaper ways to solve the problem than throwing it at £50k of hardware 

At least it's a distracting problem

Ah, the problem is lower-level than I thought... let's see if I have a handle on the fundamental issue:

                    // It's possible for an Enter thread to be deprioritized for an extended duration. It's undesirable for a
                    // deprioritized thread to keep waking up to spin despite a Sleep(1) when a large number of such threads are
                    // involved. After a threshold of Sleep(1)s, ignore the deprioritization and enter this lock to allow this
                    // thread to stop spinning and hopefully enter a proper wait state.

So basically, "extended duration" devolved into weeks/months rather than seconds due to exponential increase in recurrences from sudden increase in user loading (can't imagine where that might have come from, with EVERYBODY working from home ) which this constraint is supposed to limit but rather amplified in this case?

Because of course, silicon can do weeks of that standing on it head, while wetware get bored and nods off to sleep after a few minutes... 

In essence, your ginormous server is yet another victim of the COVID-19 crisis. 

mnem
i need more fibers in my diet.

[ArthurDent]

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) ...

So sorry to hear of your loss. I hope it helps to know that all of us here are thinking of you.

[bd139]

In essence, your ginormous server is yet another victim of the COVID-19 crisis. 

Bingo!

It doesn't help that the usual suspects here used the 1953 film Wages of Fear as a textbook.

[worsthorse]

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today 

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

I have come to the conclusion I live for the shit showers. I'm not motivated unless I am about to drown

That might explain all those Teks 

this hit closer to home than i care to admit. way too much of the "success" in my professional career was generated by my ability to repair, or make go away, very expensive catastrophes caused by people applying cookbook solutions to bespoke problems.