So about that… I actually got solar installed on my house and when the “electrician” (subbed out contractor with dubious credentials but was operating under a legit company) ran dedicated breakers for the servers… the junction box caught fire.
That was a fun 2am to wake up at.
But it’s pretty safe now. Have had several master electricians come in and evaluate it all at this point and all of them are happy with it now.
I live in a geo-stable location… Lots of DCs here specifically because of that.
house fire,
Yep. Fire extinguishers are in house, and I check them pretty regularly… Blaze cut is in the junction box(now) and in the rack. Rack is in garage, so I can just unplug and push it the fuck out (assuming I have time after throwing my kids out the windows).
It was actually my setup that alerted me to the junction fire pre-emptively… as power usage was fluctuating wildly and I have Home Assistant alert on that sort of stuff. It only sparked and blew some insulation… The fire didn’t actually move anywhere meaningful and I cut power before it had a chance to get worse. My fire alarm itself is also tied into HASS at this point. So I get alerts on my phone for the rare occasion that I leave my house (most [99%] of my work is work from home, I go to a datacenter probably quarterly at this point for all of an hour).
flood,
We barely get rain :(. But because of the soil here there’s lots of washes and irrigation mitigations in place already.
volcano,
Yellowstone finally going up will just outright kill me… don’t care to mitigate this at that point. There’s no other active volcanos around.
tornado, hurricane,
Doesn’t happen here. Dust devils are about as bad as it gets.
crashing plane,
So there’s 2 ways to look at this one… I live near an AirForce base. Either they fuck up and I’m more at risk… or because the area around the base is no-fly zone, I’m less likely… I don’t know take your pick. I would hope “less likely” due to training… but I’ve seen stupid shit when I was in the military.
theft, nazis, shitty kids, anarchists,
Guns…I have guns. Lots of ammo. A good setup for cameras on my house. And no fucks to give as army training and a deployment has forced onto me. My state is a stand your ground state. If it’s my kids that are the shitty kids… I have a backyard and a shovel (/s).
or rain?
See “flood” above.
Other sources of water (water leak): no pipes above the server rack… Water heater isn’t that close, has a freezer in between to take the brunt of any initial impact, and is brand new (so unlikely to spontaneously explode). Rack is elevated on it’s coasters and garage is graded towards the street.
We can get some interesting thunder storms here. Rack is grounded, I have backup batteries in the rack and whole house battery from the solar. But luck be what it is, in theory that could nail me as unlikely as it is.
Everything is encrypted at rest… Backups are encrypted.
I don’t have a proepr offsite yet. But my cousin is finishing building his house. He also has a fat internet pipe, and I’ll just leave a 25TB node there to backup the important/unreplaceable stuff over there. He’s clear across the country in a pretty geo-stable location as well. And he has an interest in maintaining it as he uses some services that I offer anyway (Email, nextcloud, backups, etc…). Everything else is pretty replaceable and wouldn’t take all that much effort to rebuild otherwise.
Never claimed I was “perfect”… But I’m doing pretty well here and have been doing it for nearly a decade this way. With ~30 years worth of data and very little of it lost. (there was one event over a decade ago at this point… but wasn’t all that bad.)
The real risk is just me doing something stupid since I’m the sole owner and nobody else really knows how to access any of my stuff. My dad has emergency access to my password vault, but even though he’s been programming since the 80’s, a lot of my setup is likely over his head.
All this other stuff is pretty low risk/unlikely or has been relatively decently mitigated.
Edit: Oh… and some certain important items are burned to M-discs and put into my safe every quarter or so.
So about that… I actually got solar installed on my house and when the “electrician” (subbed out contractor with dubious credentials but was operating under a legit company) ran dedicated breakers for the servers… the junction box caught fire.
That was a fun 2am to wake up at.
But it’s pretty safe now. Have had several master electricians come in and evaluate it all at this point and all of them are happy with it now.
What about things like earthquakes, house fire, flood, volcano, tornado, hurricane, crashing plane, theft, nazis, shitty kids, anarchists, or rain?
I live in a geo-stable location… Lots of DCs here specifically because of that.
Yep. Fire extinguishers are in house, and I check them pretty regularly… Blaze cut is in the junction box(now) and in the rack. Rack is in garage, so I can just unplug and push it the fuck out (assuming I have time after throwing my kids out the windows).
It was actually my setup that alerted me to the junction fire pre-emptively… as power usage was fluctuating wildly and I have Home Assistant alert on that sort of stuff. It only sparked and blew some insulation… The fire didn’t actually move anywhere meaningful and I cut power before it had a chance to get worse. My fire alarm itself is also tied into HASS at this point. So I get alerts on my phone for the rare occasion that I leave my house (most [99%] of my work is work from home, I go to a datacenter probably quarterly at this point for all of an hour).
We barely get rain :(. But because of the soil here there’s lots of washes and irrigation mitigations in place already.
Yellowstone finally going up will just outright kill me… don’t care to mitigate this at that point. There’s no other active volcanos around.
Doesn’t happen here. Dust devils are about as bad as it gets.
So there’s 2 ways to look at this one… I live near an AirForce base. Either they fuck up and I’m more at risk… or because the area around the base is no-fly zone, I’m less likely… I don’t know take your pick. I would hope “less likely” due to training… but I’ve seen stupid shit when I was in the military.
Guns…I have guns. Lots of ammo. A good setup for cameras on my house. And no fucks to give as army training and a deployment has forced onto me. My state is a stand your ground state. If it’s my kids that are the shitty kids… I have a backyard and a shovel (/s).
See “flood” above.
Other sources of water (water leak): no pipes above the server rack… Water heater isn’t that close, has a freezer in between to take the brunt of any initial impact, and is brand new (so unlikely to spontaneously explode). Rack is elevated on it’s coasters and garage is graded towards the street.
We can get some interesting thunder storms here. Rack is grounded, I have backup batteries in the rack and whole house battery from the solar. But luck be what it is, in theory that could nail me as unlikely as it is.
Everything is encrypted at rest… Backups are encrypted.
I don’t have a proepr offsite yet. But my cousin is finishing building his house. He also has a fat internet pipe, and I’ll just leave a 25TB node there to backup the important/unreplaceable stuff over there. He’s clear across the country in a pretty geo-stable location as well. And he has an interest in maintaining it as he uses some services that I offer anyway (Email, nextcloud, backups, etc…). Everything else is pretty replaceable and wouldn’t take all that much effort to rebuild otherwise.
Never claimed I was “perfect”… But I’m doing pretty well here and have been doing it for nearly a decade this way. With ~30 years worth of data and very little of it lost. (there was one event over a decade ago at this point… but wasn’t all that bad.)
The real risk is just me doing something stupid since I’m the sole owner and nobody else really knows how to access any of my stuff. My dad has emergency access to my password vault, but even though he’s been programming since the 80’s, a lot of my setup is likely over his head.
All this other stuff is pretty low risk/unlikely or has been relatively decently mitigated.
Edit: Oh… and some certain important items are burned to M-discs and put into my safe every quarter or so.