Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

  • Hacker News
  • Well.. ATC in the north of the Uk was down today as well. Fun.
  • Finding out your backup fiber has been cut, aparently some time ago, only when you try to use it as the backup isn’t an issue of “aging infrastructure.” It’s an issue of letting completely incompetent people run your tech.
  • back in the day, i worked for Energis (#3 wholesale telco in the UK) … we had about 2.2m active subscribers and ran a fibre backbone.

    one day the backbone (E1 bidirectional ring iirc) was broken north of london … my boss drove out to find that travellers had entered the field, opened the inspection cover in the ground, stood a pole for their clothes line and filled with cement

    very savvy, he didn’t let on that we had about £1m revenue per day close to collapse and convinced them to hang their clothes somewhere else (perhaps a few quid changed hands)

  • I don't quite understand this sort of thing happening. Wasn't the whole point of the internet to be a self-healing network where we route around severed cables, etc?

    Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?

  • There is also a new ATC system being rolled out, as early as today

    https://www.airwaysmag.com/new-post/faa-smart-first-deployme...

  • Building multiple diverse paths and monitoring for fiber cuts is not hard. Having 2 fiber paths is insufficient for even only moderately important workloads at a tech company. Overlapping fiber cuts happen. For something with significant economic and safety impact this is just crazy.

    Sometimes the level of incompetence / lack of care in organizations like this astounds me. I understand issues like this can be complicated and systemic but it honestly makes me think very poorly of the technologists building these systems in government.

  • > When it went to flip into the backup, we discovered that the backup fiber had a break

    Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.

    I wonder how long it was down? Days, weeks, months?

  • It happens. There use to be a joke during the first big DC build out phase that went like if you're ever going into the wilderness take a 1ft length of fiber optic cable with you. If you get lost bury it and a back hoe operator will appear and sever it within an hour. You can get a ride back with them.

Explore Birbla archives