Hot Downloads



The forum is in read only mode.
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1


Spanning-Tree Stops Trains 10 years 5 months ago #36856

  • TheBishop
  • TheBishop's Avatar Topic Author
  • Offline
  • Moderator
  • Moderator
  • Posts: 1719
  • Karma: 5
  • Thank you received: 8
Saw this today on Slashdot:

"The railway signaling failure which crippled Sydney on April 12 (some commuters reported trips of more than three hours) was caused by a failing LAN switch and software that couldn’t cope, an engineering report has found.
The switch, probably a Cisco device (Railcorp’s dominant LAN kit supplier) was part of the network in the Sydenham signaling station. That facility governs signaling for a large chunk of the Sydney rail network.
The guilty switch suffered partial failure of two electrolytic capacitors (probably the power supply). The switch is part of a dual redundant LAN which is supposed to be resilient to failure; however, the configuration couldn't handle an intermittent breakdown.
With the caps failing, the switch would shut down and try to re-start itself. This, the engineer’s report says, meant the Sydenham LAN was “caught in a cycle where it was continually trying to reconfigure itself to address the changing state of the network.”
It only took a little over ten minutes for technical staff to initiate a disaster recovery plan, but the procedure took more than an hour to complete. In that time, the software that governs the trains, known as ATRICS, was unable to cope with the flaky network. This led to a knock-on effect, taking out a system called Microloc at another station, Revesby.
With ATRICS and Microloc both failing, the rail network failed to a “safe state” in which the trains were halted where they were. Because of the hugely interdependent state of the Sydney rail network, 847 trains were delayed, 240 were cancelled, and it took the rest of April 12th for the system to recover."

Moral: We all depend on spanning-tree and assume we have resilient architectures, but spanning-tree doesn't do too well with repeated intermittent faults that come and go faster than its convergence time. The world will always need engineers...

Re: Spanning-Tree Stops Trains 10 years 5 months ago #36870

Excellent article.

... 10 years 5 months ago #36871

  • Arani
  • Arani's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 745
  • Karma: 4
  • Thank you received: 10
Informative. Just trying to grasp the domino effects starting from two electrolytic capacitors failing to 847 delays and 240 cancellations.
Picking pebbles on the shore of the networking ocean

Re: Spanning-Tree Stops Trains 10 years 5 months ago #36892

  • S0lo
  • S0lo's Avatar
  • Offline
  • Moderator
  • Moderator
  • Posts: 1577
  • Karma: 3
  • Thank you received: 7
Cisco. Lesson learned the hard way!!. Well, Let's just hope that it's learned.
Studying CCNP...

Ammar Muqaddas
Forum Moderator

Re: Spanning-Tree Stops Trains 10 years 5 months ago #36900

Very informative article
Accidents and Failures make us to change the rules
  • Page:
  • 1
Time to create page: 0.112 seconds


Cisco Routers

  • SSL WebVPN
  • Securing Routers
  • Policy Based Routing
  • Router on-a-Stick

VPN Security

  • Understand DMVPN
  • GRE/IPSec Configuration
  • Site-to-Site IPSec VPN
  • IPSec Modes

Cisco Help

  • VPN Client Windows 8
  • VPN Client Windows 7
  • CCP Display Problem
  • Cisco Support App.

Windows 2012

  • New Features
  • Licensing
  • Hyper-V / VDI
  • Install Hyper-V


  • File Permissions
  • Webmin
  • Groups - Users
  • Samba Setup