Reading that article I wonder what kind of failure that was. Did the card produce like an ARP storm or sth., or did it send noise all the time?
Thinking further I also wonder if they weren't able to detect that, wouldn't it have been possible using SNMP or sth?
I also wonder whether this shows that they a) don't have monitoring software installed and b) don't have security policies and detection systems running. It sounds like the whole system went down, makes me think about computers in the tower connected to the network and then no security system installed? Scary. But maybe (hopefully) I'm totally wrong here.
Following up on alx's comments, similar thoughts struck me when I read of how simple the actual cause was and how devastating the problem became.
First the cause - a faulty NIC. It could have been jabbering, or maybe creating a storm of noise or invalid and corrupted packets. Or, possible but less likely, a storm of broadcasts or, even less likely, a storm of valid unicast packets (although I'd have said this would be more likely down to the PC/OS/Applications than the poor old NIC). When you think on those lines, all but the last of them would have been localised by a decent switched networking infrastructure. Which leads me to wonder what their infrastructure is. Having worked in more than one environment - very recently - where the entire LAN is still unswitched due to lack of management understanding and willingness to invest...
Secondly the other popular area of management unwillingness to invest is in monitoring. And yes, with good monitoring in place a problem such as described would have been far quicker to identify and localise and the business impact would have been far less incredible.
Now I must state here that I have no knowledge of the network at LA at all, nor am I suggesting anything about their operation or criticizing them - my point is about lack of investment in our networks and how the flock only comes home to roost when there's a problem, usually from a very simple and preventable cause
Re: The Price of Economy...
11 years 9 months ago #22955