Have suggestions for a "roll your own file server"?
Ken D'Ambrosio
ken at jots.org
Wed May 26 15:34:24 EDT 2021
On 2021-05-26 12:13, Tom Buskey wrote:
> My Fedora /etc/fstab has spaces
> UUID=54103729-6e0a-4345-a2b8-8b8cded29ee1 /boot ext4
> defaults 1 2
>
> I've had clients initiate rsync for security. I think the client
> initiation would offload the rsync compute from the server.
> For a home server, it's nice to just monitor the server instead of
> multiple clients.
I'm not sure which you guys are considering client, and which server. I
like to initiate from the thing I'm backing up *to*; that way, if the
host being backed up is compromised, they won't have direct access to
the backups, themselves, which, in the days of ransomware, seems like a
valid concern. (I'd also lock down the host doing the backups pretty
tightly.)
$.02,
-Ken
> Nice buiild
>
> On Wed, May 26, 2021 at 11:00 AM Bruce Labitt
> <bruce.labitt at myfairpoint.net> wrote:
>
> Finally back to this. Built a stack of metal plates that house my
> RPI4, a boot SSD, a 1TB RAID1 array, and both active and passive USB3
> hubs. Machined parts so everything is bolted and clamped down. Have a
> PWM fan that cools the RPI4 proportional to load that runs under
> systemd. System boots from SSD. (No SD card.) It's kind of a brick
> sh!thouse, but it's sturdy. Have created the RAID1 device - or it will
> be finished in 45 minutes. It is still syncing.
>
> Now I'd like to add the md0 device to /etc/fstab. The example I see is
> with the device name. From
> https://www.tecmint.com/create-raid1-in-linux/
> /dev/md0 /mnt/raid1 ext4 defaults 0 0
>
> I've read it is better to use the UUID. Is the following the correct
> syntax?
>
> PARTUUID=my_complete_md0_UUID /mnt/raid1 ext4 defaults 0 0
>
> where my_complete_md0_UUID comes from
> $ lsblk -o UUID /dev/md0
>
> Does one need to use tabs in fstab, or are spaces ok?
>
> Once I figure this out - I have to figure out some rsync magic. Is it
> better for the server to initiate the rsync, or the remote devices?
>
> After all this I have to make another one. That shouldn't take as long
> as the first time! For some pictures of the hardware build see
> https://www.hobby-machinist.com/threads/an-rpi4-based-file-server.92273/#post-846939
>
> On 3/10/21 8:49 PM, Bruce Labitt wrote:
> I'll take a look at that. Thanks for the link.
>
> On Wed, Mar 10, 2021 at 8:15 PM Marc Nozell (marc at nozell.com)
> <nozell at gmail.com> wrote:
> Just to put a plug in for a colleague's work:
> https://perfectmediaserver.com/ It covers everything from disk
> purchasing strategies, burn-in, filesystems (ZFS, SnapRAID, etc).
>
> He also hosts a podcast that folks here may find interesting:
> https://selfhosted.show/
>
> -marc
>
> On Wed, Mar 10, 2021 at 8:08 PM <jonhall80 at comcast.net> wrote: OK:
>
> s/RPi4/some-other-cheap-computer-with-USB-3.x>/g
>
> Unless you build multiple Ethernet or WiFi or LTE modem connections
> your networking will still be the slowest thing.
>
> You do not need huge amounts of CPU power, or huge amounts of RAM.
>
> My basic point is that if you stick with simple RAID (like mirroring)
> but also set up a unit that is remote from your own home you could
> protect your own data from fire, flood and theft to a reasonable level
> and even protect your friend's data by backing up their data to your
> device.
>
> Add snapshots as suggested by Tom Buskey,perhaps encryption of file
> systems and data-streams and you can have a rather simple, server where
> you learn a lot by planning it out and setting it up rather than buying
> an "off the shelf" solution or simply using a "web backup".
>
> And good catch on the USB power supply.
>
> md
>> On 03/10/2021 6:53 PM Joshua Judson Rosen <rozzin at hackerposse.com>
>> wrote:
>>
>>
>> I'm not sure about the Raspberry Pi 4, but up thru the raspi 3+ there
>> are... problems, e.g.:
>>
>> Beware of USB on the raspi: there are some bugs in the silicon that
>> pretty severely
>> cripple performance when multiple `bulk' devices are used at
>> simultaneously,
>> sometimes to the point of making it unusable (e.g. if you want to use
>> a better Wi-Fi
>> adapter/antenna than the one built onto the board, and connect an LTE
>> modem so that
>> your raspi roam onto that if Wi-Fi becomes unavailable, throughput on
>> whichever of those
>> interfaces you're actually using can become abysmal). IIRC the issue
>> is basically
>> that the number of USB endpoints that can be assigned interrupts by
>> the raspi controller
>> is _incredibly small_; and it's common for high-throughput devices to
>> have multiple endpoints per device--
>> sometimes even one USB device will have more endpoints that the raspi
>> USB controller can handle.
>>
>> Also, `network fileserver with USB-attached hard drives' is kind of
>> the `peak unfitness'
>> for the raspberry pi. Specifically if you've got it attached to
>> ethernet,
>> the ethernet is attached through the same slow-ish USB bus as your
>> HDDs.
>>
>> (the onboard Wi-Fi BTW is SDIO; so if you avoid using the onboard
>> Wi-Fi, I guess you might also
>> be able to make your µSD card faster...)
>>
>> ALSO: you'll really want to use an externally-powered USB hub for USB
>> devices
>> that are not totally trivial, because the raspi's µUSB power supply is
>> already
>> strained... (and if you're trying to power your raspi from some random
>> USB power supply,
>> don't. Ideally you power it through the 5V pins on the expansion
>> header...).
>>
>>
>> Though there is a lot of neat stuff that can be done with a Raspberry
>> Pi,
>> it's really easy to overestimate it.
>>
>> But on the other hand: YMMV, and there are scenarios where the issues
>> don't matter,
>> and might not even be noticeable. e.g., if you're dumping periodic
>> backups to your
>> raspi asynchronously instead of (say) NFS mounting it and trying to
>> use it interactively,
>> you might not even notice the weird bottlenecks because you're never
>> looking at them.
>> And if you have enough of them as spares running simultaneously, you
>> may not care
>> that every once in a while your fileystems get corrupted or your USB
>> ports stop working
>> or whatever.
>>
>>
>> On 3/8/21 9:56 PM, jonhall80 at comcast.net wrote:
>>> I will suggest something and let people rip it apart:
>>>
>>> Get two RPis that have at least USB 2.0 Attach two large capacity
>>> disks to each one in a RAID-1 configuration (also known as
>>> "mirroring") to keep it simple. If one disk fails the other will
>>> still keep working (but you should replace it as soon as possible).
>>>
>>> Put all of your data on both systems.
>>>
>>> Take one of your systems to a friends or relatives house who you
>>> trust that has relatively good WiFi. Make sure the friend is
>>> relatively close, but is not in the same flood plain or fire area you
>>> are.
>>>
>>> Do an rsync every night to keep them in sync.
>>>
>>> Help your friend/relative do the same thing, keeping a copy of their
>>> data in your house. If your disks are big enough you could share
>>> systems and disks.
>>>
>>> Use encryption as you wish.
>>>
>>> Disk failure? Replace the disk and the data will be replicated.
>>> Fire, theft, earthquake? Take the replaced system over to your
>>> friends/relatives and copy the data at high speed, then take the
>>> copied system back to your house and start using it again.
>>>
>>> You would need three disks to fail at relatively the same time to
>>> lose your data. Or an asteroid crashing that wipes out all life on
>>> the planet. Unlikely.
>>>
>>> Realize that nothing is forever.
>>>
>>> md
>>>> On 03/08/2021 7:33 PM Bruce Labitt <bdlabitt at gmail.com> wrote:
>>>>
>>>>
>>>> For the second time in 3 months I have had a computer failure.
>>>> Oddly, it was a PS on the motherboard both times. (Two different
>>>> MB's.) Fortunately the disks were ok. I'm living on borrowed time.
>>>> Next time, I may not be that lucky.
>>>>
>>>> Need a file server system with some sort of RAID redundancy. I want
>>>> to backup 2 main computers, plus photos. Maybe this RPI4 too, since
>>>> that's what I'm running on, due to the second failure. If this SSD
>>>> goes, I'm gonna be a sad puppy. This is for home use, so we are not
>>>> talking Exabytes. I'm thinking about 2-4TB of RAID. Unless of
>>>> course, RAID is obsolete these days. Honestly, I find some of the
>>>> levels of RAID confusing. I want something that will survive a disk
>>>> failure (or two) out of the array. Have any ideas, or can you point
>>>> me to some place that discusses this somewhat intelligently?
>>>>
>>>> Are there reasonable systems that one can put together oneself these
>>>> days? Can I repurpose an older PC for this purpose? Or an RPI4?
>>>> What are the gotchas of going this way?
>>>>
>>>> I want to be able to set up a daily rsync or equivalent so we will
>>>> lose as little as possible. At the moment, I'm not thinking about
>>>> surviving fire or disaster. Maybe I should, but I suspect the costs
>>>> balloon considerably. I do not want to backup to the cloud because,
>>>> plain and simple, I don't trust it to be fully secure.
>>
>> --
>> Connect with me on the GNU social network!
>> <https://status.hackerposse.com/rozzin>
>> Not on the network? Ask me for more info!
>> _______________________________________________
>> gnhlug-discuss mailing list
>> gnhlug-discuss at mail.gnhlug.org
>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
> --
> Marc Nozell (marc at nozell.com) http://www.nozell.com/blog
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
_______________________________________________
gnhlug-discuss mailing list
gnhlug-discuss at mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
_______________________________________________
gnhlug-discuss mailing list
gnhlug-discuss at mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
_______________________________________________
gnhlug-discuss mailing list
gnhlug-discuss at mail.gnhlug.org
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210526/315393a0/attachment-0001.html
More information about the gnhlug-discuss
mailing list