A few days ago, a client’s data center (well, actually a server room) "vanished" overnight.
-
In the first sentence you mention a "data center", but such an attack would not work with a data center, to be one you need to have two buildings with independent power supply, at a safe distance, etc etc. I think this was at best a hosting room, not a data center.@uriel @stefano
worked for years for an ISP/datacenter whose primary datacenter space was in the first level of our office building. We had only one service for the building. It's technically possible to get two, but it would be from the same power company... so when the drunk driver crashed into the transformer and took out our power in winter it would have taken out both anyway. That actually caused a power surge that destroyed our transfer switch which is another problem that having two services wouldn't have solved. We did have diesel backup generators though
We didn't even have diverse entrances into the building for our fiber for a long long time either. But we were definitely a datacenter. (my brother still works there; nothing has really changed except increased bandwidth)
I have never heard of any rules or regulations that require a "datacenter" to have two buildings and independent power. Sounds like something someone made up... -
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano I wasn't aware of this kind of problems with internal monitoring and the importance of external monitoring. However, I think is more important to monitor the monitoring server or to have one heartbeat of the monitoring system (external or internal). Because the external monitoring system could also fail without being aware of it.
-
@stefano I wasn't aware of this kind of problems with internal monitoring and the importance of external monitoring. However, I think is more important to monitor the monitoring server or to have one heartbeat of the monitoring system (external or internal). Because the external monitoring system could also fail without being aware of it.
@zako sure. Monitoring the monitor is more important than monitoring the services.
-
-
@ricardo@mastodon.bsd.cafe @stefano@mastodon.bsd.cafe @mkj@social.mkj.earth No SPDs can protect you from intentional saboteurs (or faulty grid or wiring) that run hard (not momentary) 380 V into 230 V systems. Easily fry everything electrical in the building when it happens.
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
@stefano knowledge to take out a security system: aquired
-
@ricardo@mastodon.bsd.cafe @stefano@mastodon.bsd.cafe @mkj@social.mkj.earth No SPDs can protect you from intentional saboteurs (or faulty grid or wiring) that run hard (not momentary) 380 V into 230 V systems. Easily fry everything electrical in the building when it happens.
@niconiconi If my memory serves me well, a couple of years ago we installed some Schneider SPDs at a clinic in the countryside that combined types 1–3 for lightning protection
️
@stefano @mkj -
In the first sentence you mention a "data center", but such an attack would not work with a data center, to be one you need to have two buildings with independent power supply, at a safe distance, etc etc. I think this was at best a hosting room, not a data center.
@uriel Who officially defines that definition of a datacenter, I wonder?
-
A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
-
@stefano @EnigmaRotor Competing with @mwl are we?
-
@stefano @EnigmaRotor Competing with @mwl are we?
@Dianora @EnigmaRotor @mwl I could never compete with the Mentor. He's more of a spiritual guide
-
-
-
@Dianora @EnigmaRotor @mwl I could never compete with the Mentor. He's more of a spiritual guide
@stefano @Dianora @mwl yep: He Who Must Be Read is obviously untouchable. We have no rights to do that. “touch mwl” would miserably fail, that is.
Also note that Stefano’s story does not mention (yet) an overexcited systemd. I don’t see plagiarism in this. Let’s expect an hommage at some point, in pure Hans Zimmer style. /* insert big brass sound sample here */ -
@uriel Who officially defines that definition of a datacenter, I wonder?
Mostly blueprints from companies like Cisco, IBM, Google, and so on.
-
@indyradio @stefano modulating power with pulses? What is that? How does that work? What does it achieve?
I have so many questions...
Honestly, I know nothing about electrical wizzardry, I went too deep into computer science and never really touched that layer much.@pedro well, there are ways to inject noise that are malicious, that's the short story.
I think we need to pay more attention to the security of our electrical connection. -
@stefano @Dianora @mwl yep: He Who Must Be Read is obviously untouchable. We have no rights to do that. “touch mwl” would miserably fail, that is.
Also note that Stefano’s story does not mention (yet) an overexcited systemd. I don’t see plagiarism in this. Let’s expect an hommage at some point, in pure Hans Zimmer style. /* insert big brass sound sample here */@EnigmaRotor @stefano @mwl Yes mwl was careful not to ravage systemd.
-
@EnigmaRotor @stefano @mwl Yes mwl was careful not to ravage systemd.
-
Mostly blueprints from companies like Cisco, IBM, Google, and so on.
@uriel Ah, so they have their own internal guidelines as to what constitutes a datacenter, but there isn't a central definition from some kind of standards body that has coined the term with the requirement to separate buildings with independant power supplies, safe distances etc.?
-
@Dianora @EnigmaRotor @mwl I could never compete with the Mentor. He's more of a spiritual guide
@Dianora @EnigmaRotor @stefano
Spiritual guide? Your spirits are gonna go somewhere pretty dang weird, sir.
