Test Equipment Anonymous (TEA) group therapy thread

[Kosmic]

The horrible outcome is actually caused by this line of code which is courtesy microsoft. https://referencesource.microsoft.com/#system.core/system/threading/ReaderWriterLockSlim/ReaderWriterLockSlim.cs,1662 ... this is used by all "thread safe" (hahahaha) data structures. This is used as a lock implementation to aggressively cache data across all the threads to speed up a poorly written processing implementation.

Your machine is a VM with multiple VCores ?

Spinlock are notorious to be bad on VMs. Working fine on real hardware. Got a paper from Microsoft somewhere on the subject.

[bd139]

No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up

[mnementh]

I have come to the conclusion I live for the shit showers. I'm not motivated unless I am about to drown

That might explain all those Teks 

this hit closer to home than i care to admit. way too much of the "success" in my professional career was generated by my ability to repair, or make go away, very expensive catastrophes caused by people applying cookbook solutions to bespoke problems.

That is my job in a nutshell; as bd has put it before: A well-paid (by the instance) mercenary shit-shoveler. I HALO drop into somebody else's IT shitstorm with a backpack full of tools and their helldesk operator in my ear; I shovel shit furiously as directed, and hopefully together we get to the bottom (or close enough to effect a band-aid solution) before my service window closes and I have to make it through their security to my extraction point.

Shower, rinse, repeat.

bd does the same thing, only with software whereas I do it with IT hardware.


mnem
*sigh* What I wouldn't give for a nice, tedious rack & stack right now...

[Cubdriver]

BD - deepest condolences to you and yours on the loss of your FIL.  There's never a good time, and now is worse than most.

Hang in there, man...

-Pat

[bd139]

bd does the same thing, only with software whereas I do it with IT hardware.

I would love to swap my non-corporeal entities for your ones you can beat the shit out of with a hammer if it comes to it

[mnementh]

In essence, your ginormous server is yet another victim of the COVID-19 crisis. 

Bingo!

It doesn't help that the usual suspects here used the 1953 film Wages of Fear as a textbook.

*reads the synopsis*

Yeah... but balancing on that knife-edge seems to be where guys like you and me can make a living. We aren't willing to pay the price of living in the ivory tower, so we've carved out a niche for ourselves doing mercenary guerilla warfare with their inevitable shitstorms. Shower, rinse, repeat.

mnem
"It's a shitty job, but somebody's gotta do it." ~us

[mnementh]

bd does the same thing, only with software whereas I do it with IT hardware.

I would love to swap my non-corporeal entities for your ones you can beat the shit out of with a hammer if it comes to it

Yeah, but as often as I might WANT to resolve the issue with judicious application of HP P-N 19511-80014 ,  the hardware is almost NEVER the problem. 9 times out of 10, what's borked was borked by a meat-sack making a stupid mistake. 

Rather than talk someone already onsite (with the tech savvy of a sock full of lint) through plugging a laptop into a console port and firing up TeamViewer, it's quickest and most economical for them to send me to either bridge the gap between them and the nonresponsive hardware or to replace it with hardware that has been properly configured for their new fucked-up environment.

In short... as personally rewarding as it WOULD BE to perform the above-mentioned percussive maintenance on the meat-sacks responsible; they ARE the ones paying me to handle it so they don't have to, so it's kindof core to the job description that I CAN'T. 

I had one client in San Damntonio who would literally pay me for 2 or 3 visits a week for this: I would take a gateway and/or switch out of a location, replace it with another that was preconfigured. That gateway/switch would then go in my cupboard until the next such call, whereupon I'd plug it into my work PC at home, fire up TeamViewer, and Chad would remote in and configure it within an hour or two of me accepting the ticket. Shower, rinse, repeat. 

mnem
Never bring a switch to a server-fight.

[Kosmic]

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up

Yeah well imagine the mess with the new serverless trend (Ex AWS Lambda) 

[mnementh]

I think I'm going to go Virtualize me some floaty bricks with a few rounds of Beat Saber...

mnem
*~vapor~*

[Mr. Scram]

No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up

We need sandboxes in our containers in our VM's to fix the same problem thrice. Turtles all the way down, baby.

[bd139]

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up

Yeah well imagine the mess with the new serverless trend (Ex AWS Lambda) 

Argh Lambda can die in a fucking fire.

AWS is pretty damn good if you stick to the IaaS stuff but the moment you start pulling in their convenient non-portable services, which you have to do (try getting emails out of AWS without SES for example) and you're up shit creek.

No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up

We need sandboxes in our containers in our VM's to fix the same problem thrice. Turtles all the way down, baby.

That's not actually funny. These days you use Harness to run Terraform to control AWS to manage EKS which is deployed on virtual machines which run containers which contain another virtual machine that dynamically compiles an intermediate language into machine code which is running a domain specific langage to translate one form of data to another by generating another language (SQL) and sending over a network channel.



And that's why I occasionally buy things with tubes in. Corporeal entities which are possible for one human to understand and achieve things with

[bd139]

I had one client in San Damntonio who would literally pay me for 2 or 3 visits a week for this: I would take a gateway and/or switch out of a location, replace it with another that was preconfigured. That gateway/switch would then go in my cupboard until the next such call, whereupon I'd plug it into my work PC at home, fire up TeamViewer, and Chad would remote in and configure it within an hour or two of me accepting the ticket. Shower, rinse, repeat.

That's the sort of things I charge by the hour for 

[tggzzz]

No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up

We need sandboxes in our containers in our VM's to fix the same problem thrice. Turtles all the way down, baby.

Ditto asynchronous comms protocols on top of synchronous comms protocols on top of async on top of sync (repeat ad nauseum/infinitum/cretinum as appropriate).

Ditto addresses/identifiers.

I came to the conclusion that a good starting point for assessing any software design was to ask "what is an address/identifier in your system, and what are their scope and lifetime?". The amount of weebling/bumbling that provoked gave a good idea of the design (in)competence.

Similarly for any software or hardware system, asking for the definition of clock/time domains and how the boundaries are crossed, could be enlightening. Sometimes the light came on in the other person's skull, but usually it was darkness and defensive invective.

[med6753]

All these heavy and involved subjects and here I am washing windows, blinds, curtains and general Spring cleaning. Plus prepping the bedroom for painting.

I feel so insignificant.     

[Saskia]

nothing beats the excitement of your having received launch clearance by the ATC, pushed the launch button after a final look around and then watching that fucktard of a glider pilot coming out of the sun at 100 ft altitude with a speed of > 160 mph (no chance to see him in time) while your rockets launch.

No, we missed him. And he missed the NOTAM which was quite specific regarding launches in the area.

[mnementh]

I had one client in San Damntonio who would literally pay me for 2 or 3 visits a week for this: I would take a gateway and/or switch out of a location, replace it with another that was preconfigured. That gateway/switch would then go in my cupboard until the next such call, whereupon I'd plug it into my work PC at home, fire up TeamViewer, and Chad would remote in and configure it within an hour or two of me accepting the ticket. Shower, rinse, repeat.

That's the sort of things I charge by the hour for 

IT Mercenaries work by the BILLABLE Hour. We have operating expenses.

...Sometimes the light came on in the other person's skull, but usually it was darkness and defensive invective.

*Shamelessly snurches this turn of phrase for future misuse*

mnem
[ERROR: The Requested Operation Requires Elevation]

*drops stubborn workstation from top of elevator shaft*

[ERROR: Workstation Requires new CPU]

*bills client for "R&R Workstation" PLUS "HazMat detail"*

[Saskia]

pfffft- When I contracted at IBM I was able to charge by the industry minute (1 h/100). I liked it.

[mnementh]

Pffft. I round up to the next 1/4 hour; and I'm one of the nice guys. Most of my "peers" charge 30-50% more than I do, and round to the next 1/2.

Compare my measly payday to a Manufacturing Minute and it's literally just a few crumbs falling off the plate: https://ultimaker.com/learn/whats-a-manufacturing-minute-worth

mnem
"There is no line item for honor."

[Mr. Scram]

Argh Lambda can die in a fucking fire.

AWS is pretty damn good if you stick to the IaaS stuff but the moment you start pulling in their convenient non-portable services, which you have to do (try getting emails out of AWS without SES for example) and you're up shit creek.

That's not actually funny. These days you use Harness to run Terraform to control AWS to manage EKS which is deployed on virtual machines which run containers which contain another virtual machine that dynamically compiles an intermediate language into machine code which is running a domain specific langage to translate one form of data to another by generating another language (SQL) and sending over a network channel.

And that's why I occasionally buy things with tubes in. Corporeal entities which are possible for one human to understand and achieve things with

It's kind of funny we keep stacking strata of abstraction on top of each other and the solution to problems always seems to be adding more instead of backing up. Pretending these abstraction are perfect Forms without real world implications is fun and getting to the point where you outsource part of that abstraction to someone else with only a vague idea of the workings of their black box because they won't tell you is where it becomes hilarious.

[Saskia]

@mnementh I know.

Funnily enough I had a similar problem with a process running amok and holding a lock. top indicated a load of 288 and an absolutely crazy cpu time for one process.
After eastwooding the process the system settled down within a few minutes and was back to normal.

its backand is running about 700 docker containers. Which does not decrease the overall complexity ...

[tggzzz]

nothing beats the excitement of your having received launch clearance by the ATC, pushed the launch button after a final look around and then watching that fucktard of a glider pilot coming out of the sun at 100 ft altitude with a speed of > 160 mph (no chance to see him in time) while your rockets launch.

No, we missed him. And he missed the NOTAM which was quite specific regarding launches in the area.

They are damn near invisible where ever they are relative to the sun!

Makes a change from the powered pilots who overfly launchpoints below the height at which gliders release. If they hit a cable, the glider would probably be OK, the launch winch might be OK, and the powered plane would probably have its wing sawn off.

[Saskia]

problem we had with that particular glider was that you cannot abort a launch with solid propellant. There is no nozzle to close, once it burns it burns. The biggest rocket we launched at that event was about 13' and 50 lbs. if that had hit, recovery would have been impossible for the glider and a lot of people would have gotten seriously injured.

[mnementh]

Saskia = The Left Hand of Darkness Darwin. 

mnem
Don't delay! Get your award NOW!!!

[Saskia]

yeah. I was LCO (Launch Control Officer) at that event and time.
Bloody hell. That dude got into some discussion with ATC after he landed ...

[worsthorse]

All these heavy and involved subjects and here I am washing windows, blinds, curtains and general Spring cleaning. Plus prepping the bedroom for painting.

I feel so insignificant. 

I know. Whilst all this weighty talk of, well, whatever it is they are talking about was going on, I was making cookies, searing chicken for the next couple of meals, and cleaning the outdoor furniture in preparation for the arrival of sitting-on-the-deck-and-doing-nothing weather. Not a single cloud-based server to be found anywhere.