Author Topic: Test Equipment Anonymous (TEA) group therapy thread  (Read 3679012 times)

med6753, cyclin_al, Cubdriver, mnementh, JamesH, 25 CPS, andy3055, SkyMaster and 45 Guests are viewing this topic.

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55825 on: April 15, 2020, 01:06:47 pm »
So, cyclin_al, Kosmix, mnem, et al, what's the story with the somewhat recent (a year or two?) surge in TE from eBay sellers in Canada? I don't recall so many before. Are there a lot of businesses/labs upgrading or something?

Not sure. The federal government has been investing a lot in R&D. That might explain the influx of TE on ebay from public and private sources.
 
The following users thanked this post: mnementh

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55826 on: April 15, 2020, 01:07:51 pm »
Ooh nixies. I want nixies at the moment actually. There are a few nixie encrusted things here on ebay but nothing priced anything other than insanely.  :--

No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today  :-- >:(

Ugghh... when Fate takes a dump on you, she piles it on. No time for TEA?!?  |O And just to twist the knife, a death in the family AND a shitstorm at work.  :palm:   Here, have some of this:



It's green.    ;)

The other swiss army knife - Wenger. Automatic watch for size. The saw is excellent.

Rainer

And Wenger is the one that was actually Swiss Army issue, back when the name was coined. A shame they were assimilated by the mediocre collective. |O Size-wise tho, I generally prefer the feel of ~84mm scales to the larger ~91mm scales in my pocket. With a well-made pocket knife, the difference is just
peen.
  :-DD

mnem
« Last Edit: April 15, 2020, 01:12:23 pm by mnementh »
 
The following users thanked this post: Specmaster, bd139

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55827 on: April 15, 2020, 01:10:28 pm »
No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19)...

That suck  :--

All my condolences.
« Last Edit: April 15, 2020, 01:12:11 pm by Kosmic »
 
The following users thanked this post: bd139

Offline bd139

  • Super Contributor
  • ***
  • Posts: 17140
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55828 on: April 15, 2020, 01:15:41 pm »
In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing :)

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock :palm:

Threads they said. A good idea they said  :palm: :palm: :palm:

The memory is, as you mention, basically L4 cache here for the disk.
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 17140
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55829 on: April 15, 2020, 01:21:34 pm »
It's green.    ;)

Aldebaran whiskey :)
 

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55830 on: April 15, 2020, 01:29:18 pm »
So, cyclin_al, Kosmix, mnem, et al, what's the story with the somewhat recent (a year or two?) surge in TE from eBay sellers in Canada? I don't recall so many before. Are there a lot of businesses/labs upgrading or something?

Not sure. The federal government has been investing a lot in R&D. That might explain the influx of TE on ebay from public and private sources.

Mmmhmmm... might be they can see the writing on the wall and are looking to insulate their infrastructure from that of the US; maybe even build up some of their own STEM resources instead of borrowing from the neighbors. I've noticed some major changes in how my IPs are routed now vs when I first got here; even in the time since I started using my VPN. Used to be everything was routed through NYC and Jersey; now about half the time it goes through Montreal and sometimes even LA. :-//

This of course can simply be my own browsing habits, or a single bit of upgraded backbone somewhere along the way. But if my critical infrastructure were dependent upon the US not losing its collective mind, I'd certainly be looking for anywhere else in the world to be connected, at least as a backup.  :scared:

Kosmic, have you noticed any such changes, or have you always been routed primarily through Montreal since you're nor'east of me?

mnem
 :-/O
 

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55831 on: April 15, 2020, 01:34:17 pm »
It's green.    ;)

Aldebaran whiskey :)

Aye... I'm all out of Saurian Brandy.   ;)

mnem
I blame Kosmic.
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 17140
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55832 on: April 15, 2020, 01:37:14 pm »
It's green.    ;)

Aldebaran whiskey :)

Aye... I'm all out of Saurian Brandy.   ;)

mnem
I blame Kosmic.

It's good job that Romulan Ale is all I'll ever need  :-DD

 
The following users thanked this post: Specmaster

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55833 on: April 15, 2020, 01:41:29 pm »
In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing :)

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock :palm:

Threads they said. A good idea they said  :palm: :palm: :palm:

The memory is, as you mention, basically L4 cache here for the disk.

So basically too many people with authority (and not enough intelligence to use it properly) to lock portions of the db, causing resources that need to updated to be unavailable for update?

http://techgenix.com/azure-locks/

mnem
"The only limitless resource is human stupidity."
 

Offline Martin.M

  • Frequent Contributor
  • **
  • Posts: 857
  • Country: de
  • in Tek we trust
    • vintage Tek collection
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55834 on: April 15, 2020, 01:43:11 pm »
 
The following users thanked this post: Cerebus, med6753

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55835 on: April 15, 2020, 01:46:06 pm »
It's green.    ;)

Aldebaran whiskey :)

Aye... I'm all out of Saurian Brandy.   ;)

mnem
I blame Kosmic.

It's good job that Romulan Ale is all I'll ever need  :-DD


Made you laugh.   :-DD

mnem
You're welcome. ;)
« Last Edit: April 15, 2020, 01:55:16 pm by mnementh »
 
The following users thanked this post: bd139

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55836 on: April 15, 2020, 01:46:40 pm »
Kosmic, have you noticed any such changes, or have you always been routed primarily through Montreal since you're nor'east of me?

I have no idea, I never do traceroute. Might be quicker to go through Montreal since it's probably well connected with NYC.

Also depend on your provider setup. My previous provider was Teksavvy, and even though I'm in Montreal, all my traffic was routed through Toronto.
 

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
 

Offline worsthorse

  • Super Contributor
  • ***
  • Posts: 1234
  • Country: us
  • aina varma, usein väärin
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55838 on: April 15, 2020, 01:55:44 pm »

@bd139 sorry to hear about your father-in-law.  sounds like a perfect storm hitting.  i hope you work through the stuff you have some control over quickly and that the rest of it blows over without too much damage. 
specialization is for insects.
 
The following users thanked this post: bd139

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 12814
  • Country: gb
    • Having fun doing more, with less
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55839 on: April 15, 2020, 01:56:21 pm »
In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing :)

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock :palm:

Threads they said. A good idea they said  :palm: :palm: :palm:

The memory is, as you mention, basically L4 cache here for the disk.

I forgot simple incompetence repeating old errors; silly me.

Threads are fine, up to a limit of two per physical core :) Beyond that you want "job fragments" in a queue, and those are handled by one of the few worker threads. That can be extremely scalable.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 
The following users thanked this post: bd139

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 6242
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55840 on: April 15, 2020, 02:02:55 pm »
No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today  :-- >:(

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 
The following users thanked this post: mnementh, bd139

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55841 on: April 15, 2020, 02:05:19 pm »
Kosmic, have you noticed any such changes, or have you always been routed primarily through Montreal since you're nor'east of me?
I have no idea, I never do traceroute. Might be quicker to go through Montreal since it's probably well connected with NYC.

Also depend on your provider setup. My previous provider was Teksavvy, and even though I'm in Montreal, all my traffic was routed through Toronto.

Hmmm... I think Montreal has independent fiber going to London/EU; was guessing .ca might be trying to reprioritize traffic. Should be interesting to watch, especially around election time.

mnem
moo.
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 6242
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55842 on: April 15, 2020, 02:09:27 pm »
I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock :palm:

Threads they said. A good idea they said  :palm: :palm: :palm:

That's why I'm a big fan of lock-free/wait-free architectures. However, they do need the programming to be done by grown-ups who understand how to take advantage of the 'still make progress' possibilities of that architecture rather than cargo culting them and just treating them as a new kind of spin lock.
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 17140
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55843 on: April 15, 2020, 02:14:56 pm »
I forgot simple incompetence repeating old errors; silly me.

Threads are fine, up to a limit of two per physical core :) Beyond that you want "job fragments" in a queue, and those are handled by one of the few worker threads. That can be extremely scalable.

It's always the boring errors when solving boring problems.

That's technically what this is supposed to be doing. There should be one worker per core. However somewhere along the line idiocy crept in and the consumer end of the queue consumes the entire queue until there is no work spawning a thread each. It just didn't explode until the queue was busy enough for the entire machine's resources being pissed away by the cumulative spinlocks whch eventually exceed the ability for it to process work leading to a lock concurrency issue and deadlock. Which happened after it working fine for a couple of years by the looks. Obviously then this is a false assumption by people originally looking at it that "it worked fine up until now so I don't see what the issue is"  :palm: :palm: :palm:

The horrible outcome is actually caused by this line of code which is courtesy microsoft. https://referencesource.microsoft.com/#system.core/system/threading/ReaderWriterLockSlim/ReaderWriterLockSlim.cs,1662 ... this is used by all "thread safe" (hahahaha) data structures. This is used as a lock implementation to aggressively cache data across all the threads to speed up a poorly written processing implementation.

Looking at the problem it solves it was solved with a sledge hammer rather than some grey matter so I'm going to look at cheaper ways to solve the problem than throwing it at £50k of hardware  :palm: :palm:

At least it's a distracting problem :)

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock :palm:

Threads they said. A good idea they said  :palm: :palm: :palm:

That's why I'm a big fan of lock-free/wait-free architectures. However, they do need the programming to be done by grown-ups who understand how to take advantage of the 'still make progress' possibilities of that architecture rather than cargo culting them and just treating them as a new kind of spin lock.

There are a 100 lessons in your sentence there which 100 people I know have never and will never learn.

Edit: correct uncaffeinated use of words.
« Last Edit: April 15, 2020, 02:23:28 pm by bd139 »
 
The following users thanked this post: mnementh

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55844 on: April 15, 2020, 02:19:53 pm »
No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today  :-- >:(

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

No, if he were BOFH-minded he'd have known the shit-shower was coming, and offed dad-in-law (via strategic ethernet probing with HV) in advance; thereby never showing up to work at all. >:D

mnem
If you think education is expensive, try ignorance.
 
The following users thanked this post: bd139

Offline bd139

  • Super Contributor
  • ***
  • Posts: 17140
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55845 on: April 15, 2020, 02:20:45 pm »
No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today  :-- >:(

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

I have come to the conclusion I live for the shit showers. I'm not motivated unless I am about to drown :)

That might explain all those Teks  :-DD
 
The following users thanked this post: mnementh, worsthorse

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55846 on: April 15, 2020, 02:28:58 pm »
In terms of processor:X relative access times, cache is the new RAM and RAM is the new disc. You've got a bloody big disc there, and it is thrashing :)

Or perhaps the cache coherence traffic is dominating the memory traffic.

Or maybe it is the equivalent of networking bufferbloat.

So, try drastically reducing the memory size, and/or partitioning the processor/memory into independent sections.

I have located the cause. One blocking IO bound thread and about 1000 threads spinwaiting on a resource lock :palm:

Threads they said. A good idea they said  :palm: :palm: :palm:

The memory is, as you mention, basically L4 cache here for the disk.

I forgot simple incompetence repeating old errors; silly me.

Threads are fine, up to a limit of two per physical core :) Beyond that you want "job fragments" in a queue, and those are handled by one of the few worker threads. That can be extremely scalable.

Or you use Fibers  :)

Being able to suspend application code flow is a really nice and non intrusive way to implement parallelism. You end up with 1 thread per core and tons of fibers.
 
The following users thanked this post: bd139

Online mnementh

  • Super Contributor
  • ***
  • Posts: 9757
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55847 on: April 15, 2020, 02:40:01 pm »
It's always the boring errors when solving boring problems.

That's technically what this is supposed to be doing. There should be one worker per core. However somewhere along the line idiocy crept in and the consumer end of the queue consumes the entire queue until there is no work spawning a thread each. It just didn't explode until the queue was busy enough for the entire machine's resources being pissed away by the cumulative spinlocks whch eventually exceed the ability for it to process work leading to a lock concurrency issue and deadlock. Which happened after it working fine for a couple of years by the looks. Obviously then this is a false assumption by people originally looking at it that "it worked fine up until now so I don't see what the issue is"  :palm: :palm: :palm:

The horrible outcome is actually caused by this line of code which is courtesy microsoft. https://referencesource.microsoft.com/#system.core/system/threading/ReaderWriterLockSlim/ReaderWriterLockSlim.cs,1662 ... this is used by all "thread safe" (hahahaha) data structures. This is used as a lock implementation to aggressively cache data across all the threads to speed up a poorly written processing implementation.

Looking at the problem it solves it was solved with a sledge hammer rather than some grey matter so I'm going to look at cheaper ways to solve the problem than throwing it at £50k of hardware  :palm: :palm:

At least it's a distracting problem :)

Ah, the problem is lower-level than I thought... let's see if I have a handle on the fundamental issue:

Quote from: linked MS code
                    // It's possible for an Enter thread to be deprioritized for an extended duration. It's undesirable for a
                    // deprioritized thread to keep waking up to spin despite a Sleep(1) when a large number of such threads are
                    // involved. After a threshold of Sleep(1)s, ignore the deprioritization and enter this lock to allow this
                    // thread to stop spinning and hopefully enter a proper wait state.

So basically, "extended duration" devolved into weeks/months rather than seconds due to exponential increase in recurrences from sudden increase in user loading (can't imagine where that might have come from, with EVERYBODY working from home :palm:) which this constraint is supposed to limit but rather amplified in this case?

Because of course, silicon can do weeks of that standing on it head, while wetware get bored and nods off to sleep after a few minutes...  :-DD

In essence, your ginormous server is yet another victim of the COVID-19 crisis.  :o

mnem
i need more fibers in my diet.
« Last Edit: April 15, 2020, 02:43:44 pm by mnementh »
 
The following users thanked this post: bd139

Offline ArthurDent

  • Super Contributor
  • ***
  • Posts: 1193
  • Country: us
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55848 on: April 15, 2020, 02:41:20 pm »
No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) ...
So sorry to hear of your loss. I hope it helps to know that all of us here are thinking of you.
 
The following users thanked this post: mnementh, bd139

Offline bd139

  • Super Contributor
  • ***
  • Posts: 17140
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55849 on: April 15, 2020, 02:47:40 pm »
In essence, your ginormous server is yet another victim of the COVID-19 crisis.  :o

Bingo!

It doesn't help that the usual suspects here used the 1953 film Wages of Fear as a textbook.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf