Storage

What has prompted your interest in data hoarding?

Censorship and Memory-holing

I can’t tell you how many channels have disappeared and been memory-holed. Especially since censorship went into overdrive around 2019.

Data hoarders can show you how the world was before all that happened.

Unraid and Proxmox

Yes and no.

Yes if you have the resources to monitor and update. Companies have entire teams dedicated to this.

No if you don’t have the resources/time to keep up with it regularly.

IMO, no need to take this risk when you have services like Tailscale available today.

It’s not cheap to operate a business in Canada

This is very complicated to just give an answer, because:
It varies greatly based on the content. Animated compresses vastly differently than an action movie.

Varies greatly based on encoder. NEVC vs CPU etc

Varies greatly based on encoder options. I.e. -b:v -minrate -maxrate vs -rc vbr -qmin -qmaxcq values, etc

Varies greatly based on who is watching, the TV they use and their tolerance and experience.

Savings are greater at 4k than 1080p. But once you start adding HDR into the mix, you’re in a whole new world.

Even the people with very discerning eyes can’t agree on everything related to this topic. Wish I could just tell you do x… but you’ll have to test various methods and determine what you are happy with.
or, if you just want some space savings… use some default setting that cuts it in 1/2 and forget about it.

Really depends on the gear. Some of it you can, some you can’t.

Check IPMI and see if you can adjust it there. For mine, I use the IPMI plugin that you can get for Unraid. So some OS’s do have solutions. But depends on your OS and hardware.

You can replace fans, usually. But need to make sure you still have enough airflow for the equipment.

I did replace my server grade fans with the Noctua Industrial. They are still a bit loud for being in the same room, but quiet one room over. Where before you could hear it in any room of the house.
My goal was quiet one room over.

I think in general they have lower RPM and run a bit cooler and use a little less power. That usually comes with a bit less performance.

But I’m hooked on the WD Ultrastar series. Server Grade and fast. Also has low power usage, at full tilt, mine use less than 10w each. I’m running 20 hc530’s and been rock solid.

Really depends on your use case.

Unraid is wonderful and easy to use. But really has two reasons to use it:Unraid Array fits your file storage strategy. (few writes, mainly reads)You want an EZ way to get into docker and use the unraid appstore.

Other than that, you can probably find everything on Debian or Ubuntu. (I prefer Debian for services)

You can add one more:
Proxmox and then run a Debian VM for docker, for example, and compartmentalize other things you may want to run.

Also, download a mem test utility and run it overnight to test your hardware.

I started out self hosting everything… 20 years ago or so. Then I got swept up in the “cloud” movement and put so many things into the “cloud”.
Today I’m reverting back to how I started, self hosting everything that I can.

Mainly privacy, but also because they keep changing and I don’t want to have to worry about them.
I just didn’t feel like it was ‘my’ data anymore.

I like little racks

I do this with ZFS using a Keyfile and a script that runs at boot to unlock/mount.

I put the keyfiles on a USB drive. (Make sure you have backups!) This USB drive is hidden, I won’t go into details on how I did that, several ways to do that, you can get pretty creative.

If someone steals my server, they need to know where I hid my USB, or they won’t be able to get to any of the encrypted datasets.

X12SDV-4C-SP6F

Man, I have been looking into the X11SDV-8C-TP8F and I’m so close to pulling the trigger. I love the form factor.Do you know of any with 25g networking that also has EDIT: “QuickAssist” enabled like the X11SDV-8C-TP8F ? The jump from 4 core to 8 core on those adds QAT

I looked at those cases, ended up going with this because easier to expand and hte PSU size:
Can find them for less elsewhere:
https://www.amazon.com/gp/product/B095YMXW1K/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

Have a look at these HDD’s, I have seen them as low as $150. i run 20 of them right now, some have 3 years run time now. Been very happy with this vendor, usually only a few hours of runtime:
https://www.disctech.com/Western-Digital-UltraStar-DC-HC530-WUH721414ALE604-0F31156-0F31284-14TB-3.5-7.2K-RPM-512e-SATA-6Gb

Storage type:
If you need the speed, stick with ZFS.
But it should depend on the data and how risk averse you are (how easy to replace the data is). I started all ZFS, now I use this method:

Unraid Array. For easily replaceable media, like movies. (Where I have a list and can easily re-download/upload) I use only the Unraid Array for that. Mainly for data that is write once read often.
Downside:
Unraid write speed is slow. No scrubbing.
Upside:
only the 1 disk that has the data spins up. This saves me about 180-200W of power and lots of wear on the drives.
It’s also very storage efficient, I only run 2 parity disks with 20 drives. I wouldn’t do that on a regular raidz. But with unraid data lives on each disk, so you don’t loose the entire array if you lose more than 2 disks. This changes the risk math.

Then for all my critical data, that I need ZFS speed and scrubbing for, I setup a raidz1 pool with 4x Intel P4510 4TB. (Home pics/media)
Those I got for $200/pc new on eBay.

Can be safer. Can be worse.

A poorly configured self hosted vaultwarden can be a major security issue.

A properly configured one is arguable safer than hosting with a 3rd party. Lastpass taught me that one.

If you configure it to where it’s not exposed to the web, and only accessed through a VPN, like Tailscale. It can be quite robust.

That sounds easy enough, but it creates a situation where I don’t know what updates are important (security) and what updates are minor. So I have to read the release notes for each update and then decide if I need it to patch a security vulnerability.
Where with the other method, I know the update is likely critical.
For some those frequent updates are a +, for me it is not. So use what works best for you!

But right now I couldn’t use opensense even if I wanted to, as it’s FIPS non-compliant due to them still using the depreciated EOL OpenSSH 1.1.1, and no date set to move to v3

chuckle, butthurt downvotes but not one comment to dispute anything I said. Enjoy the depreciated OpenSSL without security updates.

No, I like pfsense because it has less frequent updates and is better documented.

Here is one of the better guides that helps you config much of what you are talking about:

https://nguvu.org/pfsense/pfsense-baseline-setup/

Plus, opensense gets most of their code from the work done by pfsense, and often have to wait on them to push the code. Just look at what happened with TLS 1.3

and a big part of the reason is taxes and regulations. People with $$ don’t care, but everyone in the bottom 75% really takes a big hit compared to their income.