I live in a part of the world where powercuts are pretty frequent. 1 per day is normal. They last between 1 and 8 hours. A day without powercuts feels like a special occasion.
My machine is powered by a desktop ups which is terrible. It is only supposed to power everything for a few minutes to shutdown safely. But it is cheap and I don’t know much about other affordable alternatives.
How do you folks who self host at home deal with powercuts? Any recommendations? 8 hours of uptime from a ups sounds almost impossible or totally unaffordable to me.
Ah. Texas.
I’m honestly dumbfounded how that can happen in the US. Here in Germany power outages are rare, maybe a few minutes in 8 years.
I really don’t get how something so important is left so broken in the biggest economy on earth.
I’m not trying to be mean.
Few reasons. First, the United States is huge. Texas alone is twice the size of Germany. Second, the U.S. has three main power grids. The left half, the right half, and Texas. It’s a little more complex than that, but the important part is that Texas is on its own. Third, Texas hates people. They let companies deregulate to hell and back, even at the expense of its residents.
The combination of being on its own power grid, deregulating that power grid and the companies that maintain it, and not taking proper precautions to protect its residents all leads to a less-than-reliable power grid when it gets hit with any non-standard weather. Texas especially needs to prepare for climate change, but things could definitely be going better…
Also not trying to be mean, but energy in the US is generally very cheap ($0.12/kwh where I am). With more regulation of the market, the reliability would probably increase but so would the price! (I see this happen first hand since I work in the industry)
That doesn’t sound like a bad thing tbh.
Multiply your server wattage by 8 hours. That’s how much battery you need. It’s probably not going to be a cheap investment.
The alternative would be to keep your ups and invest in a generator you can kick on if there is a power cut, but if it’s every day, that might get rough. Technology connections figured out a build it yourself solution a few years ago https://youtu.be/1q4dUt1yK0g?si=8WOTue9-zGghWlxY
I’d multiply by 10-12 hours to account for losses in the DC-AC conversion.
are there no dc-dc PSUs (or technically just voltage regulators I guess) to relace a PSU with available? That way OP could avoid part of the Ac->Dc->Ac->Dc-conversion related losses he would have with a battery-backup.
The cost of this would be much higher than just buying a bigger UPS I imagine, but it’s definitely doable from the technical standpoint
Here is an alternative Piped link(s):
https://piped.video/1q4dUt1yK0g?si=8WOTue9-zGghWlxY
Piped is a privacy-respecting open-source alternative frontend to YouTube.
I’m open-source, check me out at GitHub.
This was very informative. Thank you!
Where are you from my friend? Why do you actually need server running if you have no electricity at home? Your internet is also down right? Dont you need to just find how to shutdown safely when outage happens? Or do you have mobile/sattelite internet as a backup?
I use candles btw 🕯️
Not OP but my fiber optic Internet is not on the same power grid as the rest of my house. I’ve got a battery backup on my routers and modem for exactly this reason. I’ve got a UPS to handle a power outage into automatic graceful shutdown at 33% remaining.
Same deal here.
Can keep my WiFi up and running for hours in a power outage, even when cell signal has dropped out. Server on a separate UPS shuts down at about 40%.
That makes sense now, thx
Germany.
I don’t. Can’t remember a power outage ever except for shorting our connection box :)
Besides that only some internet outages of our ISP but that is also very rare today.
Edit: At work we sell 750VAh to our customers which are usually very small in demand and workload
OP just needs a really long extension cord plugged into a German outlet.
Another German here and yeah I also only had a single power outage (around 1h) in the last 5 or 6 years. Couldn’t even imagine having no power once every day or even every week.
for real, my homeserver in my appartment had an uptime of 450ish days before I had to power it down, because I wanted to plug in a power meter in front of it (don’t have anything fancy with redundant psus or something like that…).
and mitigating internet outages is as easy as getting a 4 or 5g router as a backup.
i stresstest my ups once a year because it doesn’t get to work otherwise 😬
I have solar panels and a backup battery for the whole house. I live in a rural area that is currently under heavy construction, as they are trying to make this area into a small city, so power outages are unfortunately extremely common.
I have a small UPS to keep my fiber and router working for a while and I have a larger UPS for my server. Even the larger UPS will only keep the server going for maybe half-an-hour, but most outages here are short. For me, the most important benefit is that my UPS will tell my server to shutdown when it begins to get short of power. Graceful shutdowns remove the risk of corruption and data loss.
I actually built my own 2 kWh battery setup after finding available commercial UPS overpriced.
It took some work and cost me about 2000 euro, but now I run everything (including networking, servers and monitor) directly on a battery feed DC net in my house.
It’s pretty cool too have all IT equipment unaffected by a power outage.
Pics!
It’s not very cleanly built, and parts of it are hidden. But this shows the main parts.
The black UPS on the left is the old one, not in use anymore.
The silver inverter on the left feed a rail in my server rack.
On the right is the battery and charger, and in the middle the fuse box and transfomer.
More info on how you built your setup please
It’s very homemade, but I believe it’s built like a DC net for a boat. It’s a bluetooth connected lithium battery, boat cabling and fuse boxes and Victron charger and voltage transformers.
I built it with “subnets” for different voltages. The battery is 24 V which feeds servers and a 34” monitor, then a transformer to 12 V for network gear, and several 5 V (USB) for a rack of raspberry pis. The is also a small 230 V transformer, for some gear that have built in PSU.
The largest server is fitted with a custom DC PSU I found on e-bay, others are normal external PSU where I cut the cables.
That’s impressive! In your experience, how does the lithium battery compare to a lead acid one?
I don’t have a good link to share, but from the research I did the difference is huge.
LiFePO4 batteries have a higher capacity, longer lifetime, safer and higher power to weight density. Many come with built in communication, like my bluetooth connection.
They are also expensive, but for my use case it’s much cheaper over time. I use about half a charge per day, which this battery should be able to sustain for 5-10 years. A lead acid battery would probably last months.
Forgot to add that a big part of the setup is in the battery controller, which I built on my own. :) That was a very fun project, and now the battery is fully automatic and charging is based on hourly price and the power provided by my solar panels.
DIY, all DC is often the way to go if you are trying to run for a long period of time. UPSs are really typically designed to run just long enough ride out brown-outs or to shut everything down safely in a total blackout. Some even shut down if they don’t sense a heavy enough load (i.e., designed to assume servers have shut down, and so preserves the battery -I banged my head against that for so long!).
I have everything on a consumer-grade APC now, and I have it set up to give me about 3 minutes of server, + another half hour of basic networking. I do have some marine deep cycles and an inverter, so I could set up the networking to run longer if cell towers were down and I needed it. But I’d likely use the energy for other things.
Can you migrate, or setup failovers, to a low powered ARM device? Or one the new Intel N series e.g. N100 low power devices?
If not, you’re going to need to buy/build a fairly large battery bank.
Yeah, been looking into Pi’s and its alternatives. But with the external drives I think I’ll need a big powerbank or I’ve to DIY a ups
What services are you running? Which of them are critical and need to stay up?
Not a lot of critical services but I would absolutely need things like pihole.
Just realized, I can host the critical ones on the ARM device and the services which I can do without for some time can stay on the current server.
Will anything even be able to use PiHole with the power out?
Then I’d go that route. Here all is on RPies, alas not the NAS, but those disks are almost always in sleep mode.
Small tip on the storage, go for a cheap SSD external (alie has a few for next to nothing), get at least 2-4, as reliability issues exists, but will show themselves within days or not. Only use rhe sd card to boot from, mount / from the ssd.
1 RPi and an ssd can runa while on a small UPS. (Need to get me one as well)
Not sure if I’m a fan of the “AliExpress SSD” recommendation. They’re badly built and unreliable, you won’t know what capacity you get, and they can be incredibly slow.
Regular, known-brand SSDs have dropped so much in price and are very affordable at low capacities. That should be a much safer investment than buying heaps of most likely unusable drives.
Depends. I used loads of ‘known brand’ micro SD cards and went trough them one per month. I ordered 2 KingSpec and ‘Kunup’ sticks, just to test. (june '20) Of each, 1 worked perfectly, 1 was not good enough for continuous use. The active Kingspec has been active for years now, but I use less then 20 GB of the 120 GB SSD. (Really need to clean up logs, OS shouldn’t use more then 5G, data is on NAS) The ones that were not reliable enough for continuous use are still in use for transport.
It worked here and proved a lot cheaper then replacing the SD card every month. As they are Chineese ‘unknown brand’, ymmv hugely. (and don’t buy something that will just fit, as trade GB isn’t IT GB and Chineese GBs vary even more) It however is always a gamble to buy something from the other side of the world. (but hey, every ‘known brand’ is made in China anyway now, so we already are hugely locked into that country)
With the price of SSDs I’d recommend an internal SSD and SATA or m.2 to USB adapter instead. That way you can choose the enclosure to provide enough cooling, and even open the adapter and add a fan if you really stress the SSD.
You can boot from the SSD directly btw
I know, and from network, but I haven’t put time into enabeling that. (and I have loads of < 1G SD cards that need to be used up anyway)
A new RPI should have USB boot enabled out of the box. I know the first year after release you had to update the firmware to get it working, but iirc that is no longer needed. Just burn the image to the stick instead of the SD card and it’s plug and play.
Look at the Turing Pi.
I live in a part of the world where powercuts are pretty frequent.
Texas?
I rarely get power outages so I just use a filesystem that can take the abuse (ei btrfs or zfs)
Any journaled filesystem is mostly fine (e.g. good old ext4).
Same as you, if power goes down for a long time I have bigger problems than not being able to access my home server. Guess I could still hook it up to my car battery and DC->AC converter if I really wanted to, and use my phone as 4G modem/backup internet access.
Ext4 can get corrupted in some cases. Its very rare but there is no way of knowing since ext4 doesn’t have file checksums.
Data centres, business, hospitals etc. run batteries to bridge the gap until the diesel starts running. It can take a minute or a few until the diesel generator takes over, but it can run for hours and days with refuelling.
Getting batteries for 8h is expensive and risky - what if the power cut suddenly lasts 9h? With batteries you have a fixed storage, with petrol or diesel you can just refuel.
Having that unreliable electricity, my home server would be the least of my problems. I would already have a generator to keep the fridge running so the food doesn’t go bad every other day.
Depending on your budget and location, a whole house backup generator can be relatively inexpensive. My family lives in a very rural area in the central US, so we have a backup whole house generator that runs on propane. I chose propane because those motors seem to have less maintenance, plus we have propane for the grill, etc, already on site.
I should probably clarify that the 8 hour ones are infrequent. Once in a month or two. But those are the days that are really annoying. The regular ones are like two hours a day or an hour at 3 different times in a day. All the other appliances are manageable but I have to shutdown my server every time.
This. Batteries are to bridge the gap until the generator is running.
Are you in South Africa? Personally I migrated to Intel NUCs and run virtualization with them. Power wise I have an Inverter and a solar panel as a backup. Inverter handles all the heavy lifting and switching. This system is purely for my electronics. So laptop, servers etc. There is no “cheap” way to do it, but if you do it in stages it can be affordable. If you can, try not to cheap out on the batteries and Inverter. Lead acid based batteries are OK IF you take care of them. Don’t use the cheapest Inverter. It’s not worth the risk of damage.
I agree. Its never worth the risk.
I think I’ll start with inverter + battery. Then add batteries in the future depending on my power needs.
There are inverters that support battery backup, recharging from solar and grid power that are supposed to go between your grid tie-in and the rest of your house. Quite a ways more expensive, but the battery capacity is probably relatively cheap compared to UPS power and is essentially a backup for your entire house.
The one I read about a while ago was a Growatt that is basically an all in one box. Can provide power from batteries, recharge from solar or grid power, feed back excess solar power to the grid, etc, you name it. And I can imagine other brands producing the same solution.
I’m lucky enough to live in a country with almost no power cuts though. I think we have at most 1 a year for max 10 minutes. So can’t say I have any experience with it myself.
If going for an inverter try a sin wave one if it’s in your budget.
some 2nd hand car batteries + 12-230v converter are probably the cheapest option
Forgive me if I’m wrong, but auto batteries aren’t meant for deep discharge. UPS’s use a specific type of battery that is meant for it.
Using auto batteries in this situation would likely end up in them dying after a few months.
“Deep cycle” batteries are the best of the lead-acids for the task. But they are still obsolete and you should source lithium if at all practical.
However if power interruptions are short, loads are low or you have an external power source like solar or wind, inferior batteries can do the job.
I use a bunch of old car batteries at my house for my battery bank. It’s more of a big capacitor, but it’s almost always sunny here and kW of solar are pouring in.
My critical equipment i.e. starlink, home and farm automation and monitoring, cell booster and HMI/SCADA only take a couple hundred watts, so no big deal. Most of the solar power goes to keeping the freezers cold.
Thanks for your insight. You seem to have some experience.
I’m currently researching a solar hybrid power system in India and am going through different battery types. We have used deep-cycle lead-acid batteries with a regular UPS and I’m familiar with their lifesoan.
Do lifepo4s last longer? I’m only seeing marginally longer lifespans. I’m also concerned about safety. I’m quite scared of regular lithiums and have read that lifepo4s are more hardy.
What about maintenance? Anything else I need to know about them?
Car batteries are cheap storage if you very rarely discharge them. You get many years if you are only using the top 80% or so of their voltage range, but if you discharge them to 50%, you only get a few hundred cycles, and if you discharge to 0%, you get dozens, if that. “Deep cycle” batteries have the same characteristic, but tend to give you more amp-ours before you hit those thresholds.
Good Lifepo4 batteries could last up to 10 years with daily full discharges. They are quite amazing in that respect. They are also likely safer than even lead acid -which need to be vented properly to avoid hydrogen gas buildup. They don’t get thermal runaway like lipos, but the cells are very much capable of producing enough current for electrical fires, so you want ones that are built properly. Maintenance is pretty much just “don’t ever charge it if it’s frozen.”
I didn’t know that deep-discharge batteries also had the same characteristics, TIL, and it makes sense based on my experience with them!
As the other commenter said, it’s all about depth of discharge. A 10kWh Lifepo4 bank gets you almost 10kWh every time while you should treat a 10kWh lead-acid bank as if it was a 2kWh bank for any sort of decent life, with deep discharges being limited to emergency situations.
All lithium chemistries are practically maintenance free while you are probably familiar with water level monitoring and equalization of lead acid.
Note that all site built lithium banks MUST have a balance mechanism as this is their “automated maintenance”. Without balancing on every charge, lithium cells will be rapidly destroyed.
Good points to note, thank you! I would have taken the balancing circuit for granted, but it doesn’t hurt to double-check with vendors.
I have 2 UPS’s, a small one that runs the fibre gear and keeps the connection alive and the main one in the rack that keeps the main server running for a couple of hours.
I’ve only ever had 1 power outage in the last 5 years though and it was scheduled electrical work. couple of brownouts during storms that were just barely deep enough to kick in the UPS boost for a minute but nothing major. nothing else is critical enough to worry about it in my case.
but if I were in a place where power is patchy, I’d have enough solar+battery for the whole house to last a normal day/night cycle, then a UPS for the rack, then a generator as a last resort only.
My setup is similar, but what I have is one UPS for the things that shouldn’t go down in case of power outage (the modem and server) and another for everything else in my room that can go down for a while without much loss (the computer, consoles, screen, etcetera)
Most desktop ups are more meant to give you time for the machine to shut down (hopefully automatically) vs actually running them for n extended period of time.
Do you have anything that would still be using the server when the power is out?
It’s not really answering your question, but are solar panels or a backup generator possible in your area? A long power outage like that everyday would be really annoying
It’s odd how people get used to the powercuts. It becomes a part of our lives.
Hadn’t thought of generators. But I feel those might be overkill for me.
