Author Topic: Test Equipment Anonymous (TEA) group therapy thread  (Read 3658901 times)

Specmaster, mansaxel, EV, Cerebus, factory, URI, Ice-Tea, paschulke2, Grandchuck, med6753, 25 CPS and 43 Guests are viewing this topic.

Offline worsthorse

  • Super Contributor
  • ***
  • Posts: 1234
  • Country: us
  • aina varma, usein väärin
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55850 on: April 15, 2020, 02:51:24 pm »
No time for TE for a few days. Unfortunately my father-in-law passed away last night (due to covid-19) and I walked into a shit show at work that's going to last days today  :-- >:(

Condolences.

Still, every cloud has a silver lining, if you were BOFH minded you could claim you need time off to deal with SWMNLBO's family at this time of crisis (probably true, even if it is with a clue by four) and thus duck the shit-shower at work.

I have come to the conclusion I live for the shit showers. I'm not motivated unless I am about to drown :)

That might explain all those Teks  :-DD

this hit closer to home than i care to admit. way too much of the "success" in my professional career was generated by my ability to repair, or make go away, very expensive catastrophes caused by people applying cookbook solutions to bespoke problems.
specialization is for insects.
 
The following users thanked this post: bd139

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55851 on: April 15, 2020, 02:53:22 pm »
The horrible outcome is actually caused by this line of code which is courtesy microsoft. https://referencesource.microsoft.com/#system.core/system/threading/ReaderWriterLockSlim/ReaderWriterLockSlim.cs,1662 ... this is used by all "thread safe" (hahahaha) data structures. This is used as a lock implementation to aggressively cache data across all the threads to speed up a poorly written processing implementation.

Your machine is a VM with multiple VCores ?

Spinlock are notorious to be bad on VMs. Working fine on real hardware. Got a paper from Microsoft somewhere on the subject.
« Last Edit: April 15, 2020, 02:56:14 pm by Kosmic »
 
The following users thanked this post: bd139

Online bd139

  • Super Contributor
  • ***
  • Posts: 17107
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55852 on: April 15, 2020, 03:03:23 pm »
No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up :)
 

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55853 on: April 15, 2020, 03:06:11 pm »
I have come to the conclusion I live for the shit showers. I'm not motivated unless I am about to drown :)

That might explain all those Teks  :-DD

this hit closer to home than i care to admit. way too much of the "success" in my professional career was generated by my ability to repair, or make go away, very expensive catastrophes caused by people applying cookbook solutions to bespoke problems.

That is my job in a nutshell; as bd has put it before: A well-paid (by the instance) mercenary shit-shoveler. I HALO drop into somebody else's IT shitstorm with a backpack full of tools and their helldesk operator in my ear; I shovel shit furiously as directed, and hopefully together we get to the bottom (or close enough to effect a band-aid solution) before my service window closes and I have to make it through their security to my extraction point.

Shower, rinse, repeat.

bd does the same thing, only with software whereas I do it with IT hardware.



mnem
*sigh* What I wouldn't give for a nice, tedious rack & stack right now...
 
The following users thanked this post: bd139

Offline Cubdriver

  • Supporter
  • ****
  • Posts: 2958
  • Country: us
  • Nixie addict
    • Photos of electronic gear
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55854 on: April 15, 2020, 03:10:58 pm »
BD - deepest condolences to you and yours on the loss of your FIL.  There's never a good time, and now is worse than most.

Hang in there, man...

-Pat
If it jams, force it.  If it breaks, you needed a new one anyway...
 
The following users thanked this post: bd139

Online bd139

  • Super Contributor
  • ***
  • Posts: 17107
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55855 on: April 15, 2020, 03:12:31 pm »
bd does the same thing, only with software whereas I do it with IT hardware.

I would love to swap my non-corporeal entities for your ones you can beat the shit out of with a hammer if it comes to it :)
 

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55856 on: April 15, 2020, 03:13:48 pm »
In essence, your ginormous server is yet another victim of the COVID-19 crisis.  :o

Bingo!

It doesn't help that the usual suspects here used the 1953 film Wages of Fear as a textbook.

*reads the synopsis*

Yeah... but balancing on that knife-edge seems to be where guys like you and me can make a living. We aren't willing to pay the price of living in the ivory tower, so we've carved out a niche for ourselves doing mercenary guerilla warfare with their inevitable shitstorms. Shower, rinse, repeat.
;)

mnem
"It's a shitty job, but somebody's gotta do it." ~us
 

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55857 on: April 15, 2020, 03:36:19 pm »
bd does the same thing, only with software whereas I do it with IT hardware.

I would love to swap my non-corporeal entities for your ones you can beat the shit out of with a hammer if it comes to it :)

Yeah, but as often as I might WANT to resolve the issue with judicious application of HP P-N 19511-80014 ,  the hardware is almost NEVER the problem. 9 times out of 10, what's borked was borked by a meat-sack making a stupid mistake.  :palm:

Rather than talk someone already onsite (with the tech savvy of a sock full of lint) through plugging a laptop into a console port and firing up TeamViewer, it's quickest and most economical for them to send me to either bridge the gap between them and the nonresponsive hardware or to replace it with hardware that has been properly configured for their new fucked-up environment.

In short... as personally rewarding as it WOULD BE to perform the above-mentioned percussive maintenance on the meat-sacks responsible; they ARE the ones paying me to handle it so they don't have to, so it's kindof core to the job description that I CAN'T.  |O

I had one client in San Damntonio who would literally pay me for 2 or 3 visits a week for this: I would take a gateway and/or switch out of a location, replace it with another that was preconfigured. That gateway/switch would then go in my cupboard until the next such call, whereupon I'd plug it into my work PC at home, fire up TeamViewer, and Chad would remote in and configure it within an hour or two of me accepting the ticket. Shower, rinse, repeat.
  ;)

mnem
Never bring a switch to a server-fight.
 
The following users thanked this post: bd139

Offline Kosmic

  • Super Contributor
  • ***
  • Posts: 1627
  • Country: ca
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55858 on: April 15, 2020, 03:38:31 pm »
I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up :)

Yeah well imagine the mess with the new serverless trend (Ex AWS Lambda)  :-DD
 
The following users thanked this post: bd139

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55859 on: April 15, 2020, 03:44:22 pm »


I think I'm going to go Virtualize me some floaty bricks with a few rounds of Beat Saber...


mnem
*~vapor~*
« Last Edit: April 15, 2020, 05:57:47 pm by mnementh »
 
The following users thanked this post: bd139

Offline Mr. Scram

  • Super Contributor
  • ***
  • Posts: 9710
  • Country: 00
  • Display aficionado
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55860 on: April 15, 2020, 03:46:18 pm »
No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up :)
We need sandboxes in our containers in our VM's to fix the same problem thrice. Turtles all the way down, baby.
 
The following users thanked this post: mnementh

Online bd139

  • Super Contributor
  • ***
  • Posts: 17107
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55861 on: April 15, 2020, 03:53:07 pm »
I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up :)

Yeah well imagine the mess with the new serverless trend (Ex AWS Lambda)  :-DD

Argh Lambda can die in a fucking fire.

AWS is pretty damn good if you stick to the IaaS stuff but the moment you start pulling in their convenient non-portable services, which you have to do (try getting emails out of AWS without SES for example) and you're up shit creek.

No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up :)
We need sandboxes in our containers in our VM's to fix the same problem thrice. Turtles all the way down, baby.

That's not actually funny. These days you use Harness to run Terraform to control AWS to manage EKS which is deployed on virtual machines which run containers which contain another virtual machine that dynamically compiles an intermediate language into machine code which is running a domain specific langage to translate one form of data to another by generating another language (SQL) and sending over a network channel.



And that's why I occasionally buy things with tubes in. Corporeal entities which are possible for one human to understand and achieve things with :)
« Last Edit: April 15, 2020, 03:54:56 pm by bd139 »
 
The following users thanked this post: BU508A, mnementh

Online bd139

  • Super Contributor
  • ***
  • Posts: 17107
  • Country: gb
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55862 on: April 15, 2020, 03:56:25 pm »
I had one client in San Damntonio who would literally pay me for 2 or 3 visits a week for this: I would take a gateway and/or switch out of a location, replace it with another that was preconfigured. That gateway/switch would then go in my cupboard until the next such call, whereupon I'd plug it into my work PC at home, fire up TeamViewer, and Chad would remote in and configure it within an hour or two of me accepting the ticket. Shower, rinse, repeat.

That's the sort of things I charge by the hour for  :-DD
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 12797
  • Country: gb
    • Having fun doing more, with less
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55863 on: April 15, 2020, 04:40:40 pm »
No bare metal. Well that node is. Most of it is Hyper-V or EKS. Aware of that one, perhaps unfortunately. The onion skin of context switching and bloated multiple kernels causes havoc particuarly on the fucked up mess that is Intel virtualization. Had serious problems with on ESX a few years back with things just hanging solid particularly when VMotion decided to move the hot nodes around while they were shitting a brick. You're only getting partial core resourcing and having to share L1/L2 cache heavily leading to high cache miss, escalating load and then deadlock. Ugh.

I'm not a proponent of virtualization. Processes are enough virtualization for anyone who hasn't fucked up :)
We need sandboxes in our containers in our VM's to fix the same problem thrice. Turtles all the way down, baby.

Ditto asynchronous comms protocols on top of synchronous comms protocols on top of async on top of sync (repeat ad nauseum/infinitum/cretinum as appropriate).

Ditto addresses/identifiers.

I came to the conclusion that a good starting point for assessing any software design was to ask "what is an address/identifier in your system, and what are their scope and lifetime?". The amount of weebling/bumbling that provoked gave a good idea of the design (in)competence.

Similarly for any software or hardware system, asking for the definition of clock/time domains and how the boundaries are crossed, could be enlightening. Sometimes the light came on in the other person's skull, but usually it was darkness and defensive invective.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 
The following users thanked this post: bd139

Online med6753

  • Super Contributor
  • ***
  • Posts: 7252
  • Country: us
  • Tek nut
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55864 on: April 15, 2020, 05:31:33 pm »
All these heavy and involved subjects and here I am washing windows, blinds, curtains and general Spring cleaning. Plus prepping the bedroom for painting.

I feel so insignificant.  :P :-DD   
An old gray beard with an attitude.
 
The following users thanked this post: Specmaster

Offline Saskia

  • Super Contributor
  • ***
  • Posts: 1168
  • Country: de
  • you unlock this door with the key of imagination
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55865 on: April 15, 2020, 06:03:38 pm »
nothing beats the excitement of your having received launch clearance by the ATC, pushed the launch button after a final look around and then watching that fucktard of a glider pilot coming out of the sun at 100 ft altitude with a speed of > 160 mph (no chance to see him in time) while your rockets launch.

No, we missed him. And he missed the NOTAM which was quite specific regarding launches in the area.

 
The following users thanked this post: mnementh, Specmaster

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55866 on: April 15, 2020, 06:11:25 pm »
I had one client in San Damntonio who would literally pay me for 2 or 3 visits a week for this: I would take a gateway and/or switch out of a location, replace it with another that was preconfigured. That gateway/switch would then go in my cupboard until the next such call, whereupon I'd plug it into my work PC at home, fire up TeamViewer, and Chad would remote in and configure it within an hour or two of me accepting the ticket. Shower, rinse, repeat.
That's the sort of things I charge by the hour for  :-DD
IT Mercenaries work by the BILLABLE Hour. We have operating expenses. ;)

...Sometimes the light came on in the other person's skull, but usually it was darkness and defensive invective.

*Shamelessly snurches this turn of phrase for future misuse* >:D

mnem
[ERROR: The Requested Operation Requires Elevation]

*drops stubborn workstation from top of elevator shaft*

[ERROR: Workstation Requires new CPU]

*bills client for "R&R Workstation" PLUS "HazMat detail"*
« Last Edit: April 15, 2020, 06:15:21 pm by mnementh »
 
The following users thanked this post: bd139

Offline Saskia

  • Super Contributor
  • ***
  • Posts: 1168
  • Country: de
  • you unlock this door with the key of imagination
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55867 on: April 15, 2020, 06:16:18 pm »
pfffft- When I contracted at IBM I was able to charge by the industry minute (1 h/100). I liked it.
 

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55868 on: April 15, 2020, 06:21:22 pm »
Pffft. I round up to the next 1/4 hour; and I'm one of the nice guys. Most of my "peers" charge 30-50% more than I do, and round to the next 1/2. :palm:

Compare my measly payday to a Manufacturing Minute and it's literally just a few crumbs falling off the plate: https://ultimaker.com/learn/whats-a-manufacturing-minute-worth

mnem
"There is no line item for honor."
« Last Edit: April 15, 2020, 06:31:50 pm by mnementh »
 
The following users thanked this post: Saskia

Offline Mr. Scram

  • Super Contributor
  • ***
  • Posts: 9710
  • Country: 00
  • Display aficionado
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55869 on: April 15, 2020, 06:40:59 pm »
Argh Lambda can die in a fucking fire.

AWS is pretty damn good if you stick to the IaaS stuff but the moment you start pulling in their convenient non-portable services, which you have to do (try getting emails out of AWS without SES for example) and you're up shit creek.

That's not actually funny. These days you use Harness to run Terraform to control AWS to manage EKS which is deployed on virtual machines which run containers which contain another virtual machine that dynamically compiles an intermediate language into machine code which is running a domain specific langage to translate one form of data to another by generating another language (SQL) and sending over a network channel.

And that's why I occasionally buy things with tubes in. Corporeal entities which are possible for one human to understand and achieve things with :)
It's kind of funny we keep stacking strata of abstraction on top of each other and the solution to problems always seems to be adding more instead of backing up. Pretending these abstraction are perfect Forms without real world implications is fun and getting to the point where you outsource part of that abstraction to someone else with only a vague idea of the workings of their black box because they won't tell you is where it becomes hilarious.
 

Offline Saskia

  • Super Contributor
  • ***
  • Posts: 1168
  • Country: de
  • you unlock this door with the key of imagination
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55870 on: April 15, 2020, 06:47:36 pm »
@mnementh I know.

Funnily enough I had a similar problem with a process running amok and holding a lock. top indicated a load of 288 and an absolutely crazy cpu time for one process.
After eastwooding the process the system settled down within a few minutes and was back to normal.

its backand is running about 700 docker containers. Which does not decrease the overall complexity ...
 
The following users thanked this post: mnementh

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 12797
  • Country: gb
    • Having fun doing more, with less
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55871 on: April 15, 2020, 06:52:12 pm »
nothing beats the excitement of your having received launch clearance by the ATC, pushed the launch button after a final look around and then watching that fucktard of a glider pilot coming out of the sun at 100 ft altitude with a speed of > 160 mph (no chance to see him in time) while your rockets launch.

No, we missed him. And he missed the NOTAM which was quite specific regarding launches in the area.

They are damn near invisible where ever they are relative to the sun!

Makes a change from the powered pilots who overfly launchpoints below the height at which gliders release. If they hit a cable, the glider would probably be OK, the launch winch might be OK, and the powered plane would probably have its wing sawn off.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline Saskia

  • Super Contributor
  • ***
  • Posts: 1168
  • Country: de
  • you unlock this door with the key of imagination
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55872 on: April 15, 2020, 06:59:27 pm »
problem we had with that particular glider was that you cannot abort a launch with solid propellant. There is no nozzle to close, once it burns it burns. The biggest rocket we launched at that event was about 13' and 50 lbs. if that had hit, recovery would have been impossible for the glider and a lot of people would have gotten seriously injured.
 

Offline mnementh

  • Super Contributor
  • ***
  • Posts: 9699
  • Country: ca
  • *Escaping The Suck*
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55873 on: April 15, 2020, 07:52:54 pm »
Saskia = The Left Hand of Darkness Darwin.  :-DD

mnem
Don't delay! Get your award NOW!!! >:D
 

Offline Saskia

  • Super Contributor
  • ***
  • Posts: 1168
  • Country: de
  • you unlock this door with the key of imagination
Re: Test Equipment Anonymous (TEA) group therapy thread
« Reply #55874 on: April 15, 2020, 08:01:03 pm »
yeah. I was LCO (Launch Control Officer) at that event and time.
Bloody hell. That dude got into some discussion with ATC after he landed ...
 
The following users thanked this post: mnementh


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf