Author Topic: Whole village broadband goes every day at 7am  (Read 2804 times)

0 Members and 1 Guest are viewing this topic.

Offline G7PSKTopic starter

  • Super Contributor
  • ***
  • Posts: 3889
  • Country: gb
  • It is hot until proved not.
Whole village broadband goes every day at 7am
« on: September 22, 2020, 03:32:02 pm »
 
The following users thanked this post: tooki

Offline Electro Fan

  • Super Contributor
  • ***
  • Posts: 3316
Re: Whole village broadband goes every day at 7am
« Reply #1 on: September 22, 2020, 04:03:00 pm »
under 4 Mbps is called a Not Spot


“Its low internet speed of 4Mbps meant it was a proper "not spot" where young people could not watch YouTube, the local pub struggled to take card payments and downloading a film was nearly impossible.”
 

Offline ChristofferB

  • Frequent Contributor
  • **
  • Posts: 929
  • Country: dk
  • Chemistry phd student!
    • My channel:
Re: Whole village broadband goes every day at 7am
« Reply #2 on: September 23, 2020, 11:39:03 am »
Amazing story. Bizarre how little radiated power is needed for a LOT of interference - on a much lower band too.
--Christoffer //IG:Chromatogiraffery
Check out my scientific instruments diy (GC, HPLC, NMR, etc) Channel: https://www.youtube.com/channel/UCZ8l6SdZuRuoSdze1dIpzAQ
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #3 on: September 23, 2020, 12:02:02 pm »
Version of this story in the Register here and original Openreach press release here.

TL;DR: "Openreach engineering, after taking over 18 months and using both hands, finally manage to find their own arse.". 

Really, what muppets put out a press release to advertise the fact that it took them more than 18 months to recognise, diagnose and track down a simple EMI problem in a tiny village? Here's the massive area that they had to roam to find this fault, and it took over 18 months:

Anybody got a syringe I can use to squeeze the magic smoke back into this?
 
The following users thanked this post: daqq, sandalcandal

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 22436
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Whole village broadband goes every day at 7am
« Reply #4 on: September 23, 2020, 12:12:26 pm »
Not to mention inability of the regulatory body (Post, is it?) to investigate.  Also surprising no amateurs complained.  Though maybe there aren't any in such a small region, and the interference was local enough not to disturb anyone else.

I wonder what the noise was.  High voltage arcing?  Must've been in a pretty unlikely location, where it was able to conduct/radiate.  Or perhaps it was a slightly newer TV, and its EMI filter went tits-up, and maybe failed isolation too, thus transmitting all the switching noise.  Wouldn't seem likely that such a malfunction would still be useful though: could it even receive cable anymore?

Also mysterious that the telecom hardware wasn't able to deal with it.  Surely they test their shite to 10V/m and such?  Maybe they don't use class A criteria (no dropout of service) during such tests.  Maybe it was installed wrong (broken shield somewhere?).

Likely, as with any good disaster, there are multiple causes of failure...

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline Kalvin

  • Super Contributor
  • ***
  • Posts: 2145
  • Country: fi
  • Embedded SW/HW.
Re: Whole village broadband goes every day at 7am
« Reply #5 on: September 23, 2020, 12:13:11 pm »
It took them 18 months because the village was tiny, and only very few people were affected. So why bother in the first place, because someone would have to leave the cozy city-office. Typical big corporate attitude *sigh*.
 

Offline sandalcandal

  • Supporter
  • ****
  • Posts: 641
  • Country: au
  • MOAR POWA!
Re: Whole village broadband goes every day at 7am
« Reply #6 on: September 23, 2020, 12:46:08 pm »
Version of this story in the Register here and original Openreach press release here.

TL;DR: "Openreach engineering, after taking over 18 months and using both hands, finally manage to find their own arse.". 

Really, what muppets put out a press release to advertise the fact that it took them more than 18 months to recognise, diagnose and track down a simple EMI problem in a tiny village? Here's the massive area that they had to roam to find this fault, and it took over 18 months:


Not really an RF or EMC person but it's my first time hearing the term Single Isolated Impulse Noise (SHINE) https://hackaday.com/2020/09/22/second-hand-television-shines-takes-down-entire-villages-internet/#comments

The name seems to imply that it is a single event that knocks off the ADSL service but it's not clear anywhere for how long; maybe just a drop then reconnect? Some comments on that hackaday article point out that the old phone lines likely used for these rural services have subpar shielding. I wonder if it was literally an old crappy degaussing system in an old CRT EMPing the local legacy copper network. Sound like a pain in the arse to find but good they *eventually* got around to fixing it.

Edit: "local engineers even replaced large sections of cable that served the village, but the problems remained."-Openreach. Nevermind that old shielded line theory sounds like they replaced the cables. How much EMI can an old crappy TV pump out?!
« Last Edit: September 23, 2020, 12:58:58 pm by sandalcandal »
Disclosure: Involved in electric vehicle and energy storage system technologies
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #7 on: September 23, 2020, 12:47:00 pm »
Not to mention inability of the regulatory body (Post, is it?) to investigate. ...

Ofcom, for details look in an English dictionary under "regulatory capture".

Quote
I wonder what the noise was.  High voltage arcing?  Must've been in a pretty unlikely location, where it was able to conduct/radiate.  Or perhaps it was a slightly newer TV, and its EMI filter went tits-up, and maybe failed isolation too, thus transmitting all the switching noise.  Wouldn't seem likely that such a malfunction would still be useful though: could it even receive cable anymore?

Cable, in a rural Welsh hamlet? I think not. US equivalent is going to be a one-horse town somewhere in the Adirondacks. [Fx: "duelling banjos" scored for harp, male choir and pit-pony to the refrain "Twll din pob sais". ] Nope, it'll most likely have been a terrestrial broadcast receiver, conceivably a satellite receiver.

The odd thing is that it can't be all that ancient as the UK stopped broadcasting analogue TV in that area in 2010 so all the truly old TVs got retired around then. In theory they could have been using an old set top box with an analogue only TV, but the generation of set top boxes designed for that mode of operation were only DVB-T and mpeg capable, now a significant number of channels have moved to DVB-T2 and X.264. So if it was of that ilk they were probably only getting half the channels being broadcast.
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 

Offline fourtytwo42

  • Super Contributor
  • ***
  • Posts: 1217
  • Country: gb
  • Interested in all things green/ECO NOT political
Re: Whole village broadband goes every day at 7am
« Reply #8 on: September 23, 2020, 01:02:33 pm »
The situation regarding interference here in the UK is dire, probably as a result of de-regulation nobody is interested.
I have had severe broadband interference for years, knocking out the link and causing poor performance due to the exchange enforcing higher snr's to cope with the noise. I was also able to find it on AM radio blotting out a BBC (national broadcaster) channel however the BBC despite there claims to investigate refused pointing me at the regulator (OFCOM) who also refused pointing me at the BBC. After several circuits of this I have up. Eventually I cobbled together an RDF set and traced it to a local business using crap Chinese electronic starters for there florescent lighting but unlike the gentleman with the TV in this thread have continued using them despite being shown the problem.
I did at least manage to get open reach to replace the old figure 8 drop wire with a modern twisted variety and route it away from the power cables along the gables of the house and that has reduced the coupling a bit. Of course those in the village who have dumb green light routers are unaware of this problem, just fed up with very low speeds!
Eventually I will either pay extra for fibre (FTTP) or move house.
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #9 on: September 23, 2020, 01:15:20 pm »
Version of this story in the Register here and original Openreach press release here.

TL;DR: "Openreach engineering, after taking over 18 months and using both hands, finally manage to find their own arse.". 

Really, what muppets put out a press release to advertise the fact that it took them more than 18 months to recognise, diagnose and track down a simple EMI problem in a tiny village? Here's the massive area that they had to roam to find this fault, and it took over 18 months:


Not really an RF or EMC person but it's my first time hearing the term Single Isolated Impulse Noise (SHINE) https://hackaday.com/2020/09/22/second-hand-television-shines-takes-down-entire-villages-internet/#comments

The name seems to imply that it is a single event that knocks off the ADSL service but it's not clear anywhere for how long; maybe just a drop then reconnect? Some comments on that hackaday article point out that the old phone lines likely used for these rural services have subpar shielding. I wonder if it was literally an old crappy degaussing system in an old CRT EMPing the local legacy copper network.

The kicker in this is that they wholesale replaced all the local cabling for the whole village, and then took a whole further 18 months to track the fault down. Don't go looking for subtle explanations when anybody who has had to deal with BT/Openreach engineering can tell you that the problem is a fundamental lack of competence and training. Openreach 'engineers' get the bare minimum training, strictly what is required to do their prescribed tasks and they rarely come from a background with boarder electronics/comms knowledge. I had an 'engineer' out for a fault who didn't understand the term "white noise". There are (or at least were) pockets of excellence in that organisation, but they're kept well away from customers.

The problem was a regular, daily, timed event for months and months and nobody, but nobody, went "Well it's obviously to do with something getting turned on/off at 7AM and radiating EMI; let's look for a big motor being turned on around about then, and if we can't find one, let's do a field survey at 7AM.". Or if they did, nobody listened to them. No, it took a 'crack team' from headquarters to turn up with a field survey spectrum analyser and it took 18 months to call them in.

What gets me is that the organization is dumb enough to put this out as a success story, a "We're so good and persistent because we fixed a problem that was so hard to find it took 18 months, but we persevered" when to anybody who understands, the message is "It was a simple problem and we were too incompetent to fix it for 18 months". It's a failure on two fronts, incompetence at finding and fixing the fault, and then incompetence in trumpeting a failure as a success. Anyone who has a brain, and experience with large corporations, will recognise the syndrome at work here.
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 
The following users thanked this post: fourtytwo42, sandalcandal

Offline rdl

  • Super Contributor
  • ***
  • Posts: 3667
  • Country: us
Re: Whole village broadband goes every day at 7am
« Reply #10 on: September 23, 2020, 01:32:01 pm »
I didn't think there could be an internet provider more useless and incompetent than Comcast, but apparently I was wrong.
 

Offline madires

  • Super Contributor
  • ***
  • Posts: 8274
  • Country: de
  • A qualified hobbyist ;)
Re: Whole village broadband goes every day at 7am
« Reply #11 on: September 23, 2020, 01:43:43 pm »
The kicker in this is that they wholesale replaced all the local cabling for the whole village, and then took a whole further 18 months to track the fault down.

Ouch! Experts with dangerous half-knowledge at work?
 

Offline sandalcandal

  • Supporter
  • ****
  • Posts: 641
  • Country: au
  • MOAR POWA!
Re: Whole village broadband goes every day at 7am
« Reply #12 on: September 23, 2020, 01:55:56 pm »
The kicker in this is that they wholesale replaced all the local cabling for the whole village, and then took a whole further 18 months to track the fault down. Don't go looking for subtle explanations when anybody who has had to deal with BT/Openreach engineering can tell you that the problem is a fundamental lack of competence and training. Openreach 'engineers' get the bare minimum training, strictly what is required to do their prescribed tasks and they rarely come from a background with boarder electronics/comms knowledge. I had an 'engineer' out for a fault who didn't understand the term "white noise". There are (or at least were) pockets of excellence in that organisation, but they're kept well away from customers.

Yikes, didn't think it would be that bad.

I read that openreach press release and edited my post 15 min before Cerebus' reply for the record.
Disclosure: Involved in electric vehicle and energy storage system technologies
 

Offline nali

  • Frequent Contributor
  • **
  • Posts: 732
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #13 on: September 23, 2020, 02:01:41 pm »
BT Openreach are a bit of a mixed bunch. Like the person who replaced the copper line to my house as scheduled preventative maintenance while I was at work - it was terminated at a junction box in my roofline which actually had 2 sets of twisted pairs (one was an old redundant 2nd fax line no longer connected). Yes, he connected the incoming line to the wrong pair.

Then after I reported the loss of service (at this point I had no idea the line had been replaced) the same engineer came back to test and I was told "tested OK to customer premises". He'd attached a test set outside at the junction box he'd just wrongly rewired.

Day 3 I stayed home while another engineer came, spotted the issue straight away and was pretty angry that they'd been made to look like a bunch of twats and promised he'd "have a word" when he got back.
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #14 on: September 23, 2020, 02:18:19 pm »
Day 3 I stayed home while another engineer came, spotted the issue straight away and was pretty angry that they'd been made to look like a bunch of twats and promised he'd "have a word" when he got back.

I get the impression that every area has a bunch of trained monkeys, and a few guys who are interested and understand what they're doing. First visit you whom get is pop luck, second visit ditto, third visit the local supervisor bends the system a bit and assigns someone from the 'known good' category, probably people with a high fix rate who when they mark a job "tested and working" it actually means "tested and working" not "send another guy in three days if the root cause is anything vaguely out of the ordinary". As in all large organizations that are run on procedures, processes, rules and the like there are people who keep out of trouble by just following the procedure and there are people who actually do their job.
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 

Offline G7PSKTopic starter

  • Super Contributor
  • ***
  • Posts: 3889
  • Country: gb
  • It is hot until proved not.
Re: Whole village broadband goes every day at 7am
« Reply #15 on: September 23, 2020, 02:53:12 pm »
Is it possible that the broadband in the village is via a radio link system with a central mast. Some isolated areas are hooked up like that. Also it must have been a continuous signal as turning the TV off solved the problem instantly, may have a large burst followed by continuous transmission. Otherwise they would not have known until the following day if the problem was fixed.
 

Offline Gyro

  • Super Contributor
  • ***
  • Posts: 10172
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #16 on: September 23, 2020, 02:59:46 pm »
I get the impression that every area has a bunch of trained monkeys, and a few guys who are interested and understand what they're doing. First visit you whom get is pop luck, second visit ditto, third visit the local supervisor bends the system a bit and assigns someone from the 'known good' category, probably people with a high fix rate who when they mark a job "tested and working" it actually means "tested and working" not "send another guy in three days if the root cause is anything vaguely out of the ordinary". As in all large organizations that are run on procedures, processes, rules and the like there are people who keep out of trouble by just following the procedure and there are people who actually do their job.

They have different levels of staff and an escalation procedure (or so the last Openreach 'troubleshooter' told me when we had a persistent problem a few years back). It makes economic sense (for Openreach). They don't want your more experienced, and presumably better paid guys bogged down with the daily crud that lower skilled staff can do, they'd get fed up and leave anyway.

Most companies have some sort of escalation procedure, whatever their activity - Unless it's Sky or Virgin of course, they're universally crap.  >:D
Best Regards, Chris
 

Offline Electro Fan

  • Super Contributor
  • ***
  • Posts: 3316
Re: Whole village broadband goes every day at 7am
« Reply #17 on: September 23, 2020, 03:03:25 pm »
Day 3 I stayed home while another engineer came, spotted the issue straight away and was pretty angry that they'd been made to look like a bunch of twats and promised he'd "have a word" when he got back.

I get the impression that every area has a bunch of trained monkeys, and a few guys who are interested and understand what they're doing. First visit you whom get is pop luck, second visit ditto, third visit the local supervisor bends the system a bit and assigns someone from the 'known good' category, probably people with a high fix rate who when they mark a job "tested and working" it actually means "tested and working" not "send another guy in three days if the root cause is anything vaguely out of the ordinary". As in all large organizations that are run on procedures, processes, rules and the like there are people who keep out of trouble by just following the procedure and there are people who actually do their job.

Pretty much this ^ exactly. 

It seems almost endemic in the telecom industry.

Unfortunately the typical telecom company environment becomes so mindlessly rigid that little incentive remains for anyone to fix the environment.  As a result customers are forced to choose between accepting a persistent problem or spending inordinate amounts of time to get the problem addressed and resolved when if any of the three was working properly among the triumvirate of “people, process, and technology” the problem would have been either prevented or readily fixed.
 

Online Ian.M

  • Super Contributor
  • ***
  • Posts: 13216
Re: Whole village broadband goes every day at 7am
« Reply #18 on: September 23, 2020, 03:04:48 pm »
I didn't think there could be an internet provider more useless and incompetent than Comcast, but apparently I was wrong.
What's worse, BT/Openreach provide the infrastructure to *ALL* UK ISPs that offer (A)DSL service over POTS outside the Hull Telecoms area (the last surviving independent municipal phone company), so its virtually impossible to get  broadband without using their services.  Even if you sign up with a cable company or for mobile data, there's a very high probability Openreach will be involved with the backhaul.   
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #19 on: September 23, 2020, 03:35:16 pm »
I get the impression that every area has a bunch of trained monkeys, and a few guys who are interested and understand what they're doing. First visit you whom get is pop luck, second visit ditto, third visit the local supervisor bends the system a bit and assigns someone from the 'known good' category, probably people with a high fix rate who when they mark a job "tested and working" it actually means "tested and working" not "send another guy in three days if the root cause is anything vaguely out of the ordinary". As in all large organizations that are run on procedures, processes, rules and the like there are people who keep out of trouble by just following the procedure and there are people who actually do their job.

They have different levels of staff and an escalation procedure (or so the last Openreach 'troubleshooter' told me when we had a persistent problem a few years back). It makes economic sense (for Openreach). They don't want your more experienced, and presumably better paid guys bogged down with the daily crud that lower skilled staff can do, they'd get fed up and leave anyway.

Most companies have some sort of escalation procedure, whatever their activity - Unless it's Sky or Virgin of course, they're universally crap.  >:D

Yes and no. I'm talking about specifically the first line local guys and the variability between them. You'll know if you've been escalated up to the next layer, it's a quite explicit process. (Says a man who, in the pre-openreach it's all BT days, once had a fault escalated all the way to the chairman's office, which has/had a specialist fault management team, where it still sat for around 3 months before it got resolved. At one point in time it felt as if my life was 50% managing BT's management of faults.)
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 

Offline Cerebus

  • Super Contributor
  • ***
  • Posts: 10576
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #20 on: September 23, 2020, 03:48:02 pm »
I didn't think there could be an internet provider more useless and incompetent than Comcast, but apparently I was wrong.
What's worse, BT/Openreach provide the infrastructure to *ALL* UK ISPs that offer (A)DSL service over POTS outside the Hull Telecoms area (the last surviving independent municipal phone company), so its virtually impossible to get  broadband without using their services.  Even if you sign up with a cable company or for mobile data, there's a very high probability Openreach will be involved with the backhaul.

No, in some areas TalkTalk have their own infrastructure. There the only Openreach infrastructure involved is the copper from the customer premises to either a local TalkTalk street cabinet (co-located next to an openreach street cabinet) or to a TalkTalk owned and operated DSLAM at the local exchange.

Very very rarely, in very few places, there are other smaller operators who do the same as TalkTalk or even run their own copper to the premises. Where I'm living now, the premises are served by Openreach and (former) Cable and Wireless* copper, and my previous place had copper service from BT and Telewest (Telewest merged with NTL and then in turn with Virgin). TalkTalk are however the only player of any significant size in this part of the market.

* The copper side of C&W's business was sold to Vodafone, who then sold it to Virgin/Liberty Global (from memory).
Anybody got a syringe I can use to squeeze the magic smoke back into this?
 

Online coppice

  • Super Contributor
  • ***
  • Posts: 10031
  • Country: gb
Re: Whole village broadband goes every day at 7am
« Reply #21 on: September 23, 2020, 04:01:30 pm »
Really, what muppets put out a press release to advertise the fact that it took them more than 18 months to recognise, diagnose and track down a simple EMI problem in a tiny village? Here's the massive area that they had to roam to find this fault, and it took over 18 months:
At a guess I'd say someone who wanted to embarrass their management into allowing them to use more effective practices. I've been through similar things. We had a line fault that showed up every time there was rain, but had disappeared by the time anyone came to check. The rules the technicians worked under would not allow them to move us to one of the many alternative lines available. After a number of visits a technician asked us to cooperate in the only fix he knew worked. He came, left and we immediately called to complain, regardless of whether the line was working. This repeated every few hours, with him arriving just to tell us to make the next call. It took about THREE WEEKS of this before his work sheet finally told him to change the line.
 

Offline rdl

  • Super Contributor
  • ***
  • Posts: 3667
  • Country: us
Re: Whole village broadband goes every day at 7am
« Reply #22 on: September 23, 2020, 04:11:24 pm »
Once when my internet stopped working, it took Comcast 3 days and a technician visit to figure out they had terminated service to the wrong apartment.
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 16385
  • Country: za
Re: Whole village broadband goes every day at 7am
« Reply #23 on: September 23, 2020, 04:14:01 pm »
They replaced cables, a far cry from the South African Telkom, who are abandoning copper as "uneconomical", and are as busy as possible trying to roll out fibre instead. Yet still in the process of racing to be the smallest player, a far cry from a quarter century ago when they were the monopoly, and have been made a has been by the march of technology, and them refusing to give up the cash cow of infrastructure copper.

You needed a phone line to get DSL, no local loop unbundling till later, and no naked DSL till this year, just when the announcement came they are exiting the copper line market. Lost out because they would limit line speed down unless you paid for the higher speeds, and would never allow naked DSL, despite more than half the phone lines using DSL did not have a phone attached. Slow to bring in fibre, and eclipsed by the mobile operators and the wireless providers, at least till their stranglehold and monopoly on long links was broken, along with international carrier traffic.

Report a fault and that 18 month wait might be for them to respond to you with a single reply, and actually resolving the problem is a toss up of them fixing it by changing the port on the DSLAM, rolling the fault to somebody else, or them saying "it is a cable fault, handing off to outdoor tech" and you are in another 19 month wait, but still are required to pay for the line in the interim, unless you cancel the service outright, and good luck getting the refund.
 

Online Ian.M

  • Super Contributor
  • ***
  • Posts: 13216
Re: Whole village broadband goes every day at 7am
« Reply #24 on: September 23, 2020, 04:22:21 pm »
I'm glad to have been corrected.  *ANY* competition to BT's copper near-monopoly is a very good thing.   However, IIRC TalkTalk have historically have had a less than stellar reputation, so I'm not sure they are significantly preferable to being beholden to Openreach.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf