A few days ago, a client’s data center (well, actually a server room) "vanished" overnight.

A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.

I then suspected a power failure, but the UPS should have sent an alert.

The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.

To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.

The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.

That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.

The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.

The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.

Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.

Never rely only on internal monitoring. Never.

#IT #SysAdmin #HorrorStories #ITHorrorStories #Monitoring

@stefano Good for you. If next time, you could solve your problems without involving people who are sick at home with a serious family issue on top, that would be great.

A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.

I then suspected a power failure, but the UPS should have sent an alert.

The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.

To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.

The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.

That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.

The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.

The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.

Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.

Never rely only on internal monitoring. Never.

#IT #SysAdmin #HorrorStories #ITHorrorStories #Monitoring

@paul @fennek I am familiar with that organization - and I know that the person (the one who was home sick, even if I didn't know he was home sick) has a deep sense of loyalty, but he is not reckless. If he hadn't been well enough, he wouldn't have gone. I even offered to go in his place myself. It is a healthy environment, not "that typical company" that exploits its employees. For obvious reasons, I cannot disclose details (and I work with several similar companies in different areas), but I can guarantee that everyone acted with the utmost respect for human decency. Fortunately, not all businesses operate like malicious entities that only think about harming their employees and collaborators.

I always strive to distance myself from such organizations, as they do not align with my outlook on life and the world.

@stefano Thank you for the clarification.

@stefano @gumnos @mwl @Dianora Sure, inhabited by a Bored Single Demon. “He” just wanted to get friends.

@mwl @EnigmaRotor @stefano Was that an offer to buy us all a round of beers at BSDCan? *whistles innocently*

@Dianora @mwl @EnigmaRotor @stefano try Gelato, just sayin'

@Keltounet @mwl @EnigmaRotor @stefano Grand Marnier Gelato!

@Keltounet @mwl @EnigmaRotor @stefano Grand Marnier Gelato!

@Dianora @Keltounet @mwl @EnigmaRotor ok, that's settled - Grand Marnier Gelato in Ottawa, in June

@stefano @Dianora @Keltounet @mwl I’ll virtually enjoy from remote… (well, no, but that’s for another conf a bit closer from home, someday). (I don’t think gelato payloads can make it to my computer, probably due to firewalls, would they?).

@stefano @Dianora @Keltounet @mwl I might as well purchase vanilla icecream and put a large amount of Grand Marnier on it. Call it emulation of some sort , and try to synchronize with you folks. Well, timezones would make this fun, indeed.

@EnigmaRotor @Dianora @Keltounet @mwl that should work

@stefano @Dianora @Keltounet @mwl You know the trade offs of emulation, this could lead to a suboptimal experience… thus I might get rid of the icecream and focus on the bottle. (From simplicity comes user satisfaction)

#drunkCipherMachine #veryRandomNumberGenerator

@stefano @Dianora @Keltounet @mwl You know the trade offs of emulation, this could lead to a suboptimal experience… thus I might get rid of the icecream and focus on the bottle. (From simplicity comes user satisfaction)

#drunkCipherMachine #veryRandomNumberGenerator

@stefano @Dianora @Keltounet @mwl Then I might loose control and replace the bottle with génépi, which would further deviate from the original gelato product but amplify even user satisfaction to a whole new level. #outofcontrol #emergencyStop #forgetPreviousPrompt

@stefano @Dianora @Keltounet @mwl Footnote : https://fr.wikipedia.org/wiki/G%C3%A9n%C3%A9pi
I wish this article would be available in English too, yet it is available in Italian. (For some reason)

@EnigmaRotor @Dianora @Keltounet @mwl yes, for very good reasons