Have suggestions for a "roll your own file server"?

Wed May 26 12:13:20 EDT 2021

My Fedora /etc/fstab has spaces
UUID=54103729-6e0a-4345-a2b8-8b8cded29ee1 /boot                   ext4
 defaults        1 2

I've had clients initiate rsync for security.  I think the client
initiation would offload the rsync compute from the server.
For a home server, it's nice to just monitor the server instead of multiple
clients.

Nice buiild

On Wed, May 26, 2021 at 11:00 AM Bruce Labitt <bruce.labitt at myfairpoint.net>
wrote:

> Finally back to this.  Built a stack of metal plates that house my RPI4, a
> boot SSD, a 1TB RAID1 array, and both active and passive USB3 hubs.
> Machined parts so everything is bolted and clamped down.  Have a PWM fan
> that cools the RPI4 proportional to load that runs under systemd.  System
> boots from SSD.  (No SD card.)  It's kind of a brick sh!thouse, but it's
> sturdy.  Have created the RAID1 device - or it will be finished in 45
> minutes.  It is still syncing.
>
> Now I'd like to add the md0 device to /etc/fstab.  The example I see is
> with the device name.  From https://www.tecmint.com/create-raid1-in-linux/
> /dev/md0    /mnt/raid1    ext4    defaults    0 0
>
> I've read it is better to use the UUID.  Is the following the correct
> syntax?
>
> PARTUUID=my_complete_md0_UUID  /mnt/raid1    ext4    defaults  0 0
>
> where my_complete_md0_UUID comes from
> $ lsblk -o UUID /dev/md0
>
> Does one need to use tabs in fstab, or are spaces ok?
>
> Once I figure this out - I have to figure out some rsync magic.  Is it
> better for the server to initiate the rsync, or the remote devices?
>
> After all this I have to make another one.  That shouldn't take as long as
> the first time!  For some pictures of the hardware build see
> https://www.hobby-machinist.com/threads/an-rpi4-based-file-server.92273/#post-846939
>
>
>
> On 3/10/21 8:49 PM, Bruce Labitt wrote:
>
> I'll take a look at that.  Thanks for the link.
>
> On Wed, Mar 10, 2021 at 8:15 PM Marc Nozell (marc at nozell.com) <
> nozell at gmail.com> wrote:
>
>> Just to put a plug in for a colleague's work:
>> https://perfectmediaserver.com/    It covers everything from disk
>> purchasing strategies, burn-in, filesystems (ZFS, SnapRAID, etc).
>>
>> He also hosts a podcast that folks here may find interesting:
>> https://selfhosted.show/
>>
>> -marc
>>
>> On Wed, Mar 10, 2021 at 8:08 PM <jonhall80 at comcast.net> wrote:
>>
>>> OK:
>>>
>>> s/RPi4/some-other-cheap-computer-with-USB-3.x>/g
>>>
>>> Unless you build multiple Ethernet or WiFi or LTE modem connections your
>>> networking will still be the slowest thing.
>>>
>>> You do not need huge amounts of CPU power, or huge amounts of RAM.
>>>
>>> My basic point is that if you stick with simple RAID (like mirroring)
>>> but also set up a unit that is remote from your own home you could protect
>>> your own data from fire, flood and theft to a reasonable level and even
>>> protect your friend's data by backing up their data to your device.
>>>
>>> Add snapshots as suggested by Tom Buskey,perhaps encryption of file
>>> systems and data-streams and you can have a rather simple, server where you
>>> learn a lot by planning it out and setting it up rather than buying an "off
>>> the shelf" solution or simply using a "web backup".
>>>
>>> And good catch on the USB power supply.
>>>
>>> md
>>> > On 03/10/2021 6:53 PM Joshua Judson Rosen <rozzin at hackerposse.com>
>>> wrote:
>>> >
>>> >
>>> > I'm not sure about the Raspberry Pi 4, but up thru the raspi 3+ there
>>> are... problems, e.g.:
>>> >
>>> > Beware of USB on the raspi: there are some bugs in the silicon that
>>> pretty severely
>>> > cripple performance when multiple `bulk' devices are used at
>>> simultaneously,
>>> > sometimes to the point of making it unusable (e.g. if you want to use
>>> a better Wi-Fi
>>> > adapter/antenna than the one built onto the board, and connect an LTE
>>> modem so that
>>> > your raspi roam onto that if Wi-Fi becomes unavailable, throughput on
>>> whichever of those
>>> > interfaces you're actually using can become abysmal). IIRC the issue
>>> is basically
>>> > that the number of USB endpoints that can be assigned interrupts by
>>> the raspi controller
>>> > is _incredibly small_; and it's common for high-throughput devices to
>>> have multiple endpoints per device--
>>> > sometimes even one USB device will have more endpoints that the raspi
>>> USB controller can handle.
>>> >
>>> > Also, `network fileserver with USB-attached hard drives' is kind of
>>> the `peak unfitness'
>>> > for the raspberry pi. Specifically if you've got it attached to
>>> ethernet,
>>> > the ethernet is attached through the same slow-ish USB bus as your
>>> HDDs.
>>> >
>>> > (the onboard Wi-Fi BTW is SDIO; so if you avoid using the onboard
>>> Wi-Fi, I guess you might also
>>> >  be able to make your µSD card faster...)
>>> >
>>> > ALSO: you'll really want to use an externally-powered USB hub for USB
>>> devices
>>> > that are not totally trivial, because the raspi's µUSB power supply is
>>> already
>>> > strained... (and if you're trying to power your raspi from some random
>>> USB power supply,
>>> > don't. Ideally you power it through the 5V pins on the expansion
>>> header...).
>>> >
>>> >
>>> > Though there is a lot of neat stuff that can be done with a Raspberry
>>> Pi,
>>> > it's really easy to overestimate it.
>>> >
>>> > But on the other hand: YMMV, and there are scenarios where the issues
>>> don't matter,
>>> > and might not even be noticeable. e.g., if you're dumping periodic
>>> backups to your
>>> > raspi asynchronously instead of (say) NFS mounting it and trying to
>>> use it interactively,
>>> > you might not even notice the weird bottlenecks because you're never
>>> looking at them.
>>> > And if you have enough of them as spares running simultaneously, you
>>> may not care
>>> > that every once in a while your fileystems get corrupted or your USB
>>> ports stop working
>>> > or whatever.
>>> >
>>> >
>>> > On 3/8/21 9:56 PM, jonhall80 at comcast.net wrote:
>>> > > I will suggest something and let people rip it apart:
>>> > >
>>> > > Get two RPis that have at least USB 2.0  Attach two large capacity
>>> disks to each one in a RAID-1 configuration (also known as "mirroring") to
>>> keep it simple.  If one disk fails the other will still keep working (but
>>> you should replace it as soon as possible).
>>> > >
>>> > > Put all of your data on both systems.
>>> > >
>>> > > Take one of your systems to a friends or relatives house who you
>>> trust that has relatively good WiFi.  Make sure the friend is relatively
>>> close, but is not in the same flood plain or fire area you are.
>>> > >
>>> > > Do an rsync every night to keep them in sync.
>>> > >
>>> > > Help your friend/relative do the same thing, keeping a copy of their
>>> data in your house.   If your disks are big enough you could share systems
>>> and disks.
>>> > >
>>> > > Use encryption as you wish.
>>> > >
>>> > > Disk failure?   Replace the disk and the data will be replicated.
>>> > > Fire, theft, earthquake?   Take the replaced system over to your
>>> friends/relatives and copy the data at high speed, then take the copied
>>> system back to your house and start using it again.
>>> > >
>>> > > You would need three disks to fail at relatively the same time to
>>> lose your data.   Or an asteroid crashing that wipes out all life on the
>>> planet.  Unlikely.
>>> > >
>>> > > Realize that nothing is forever.
>>> > >
>>> > > md
>>> > >> On 03/08/2021 7:33 PM Bruce Labitt <bdlabitt at gmail.com> wrote:
>>> > >>
>>> > >>
>>> > >> For the second time in 3 months I have had a computer failure.
>>> Oddly, it was a PS on the motherboard both times.  (Two different MB's.)
>>> Fortunately the disks were ok.  I'm living on borrowed time.  Next time, I
>>> may not be that lucky.
>>> > >>
>>> > >> Need a file server system with some sort of RAID redundancy.
>>> I want to backup 2 main computers, plus photos.  Maybe this RPI4 too, since
>>> that's what I'm running on, due to the second failure.  If this SSD goes,
>>> I'm gonna be a sad puppy.  This is for home use, so we are not talking
>>> Exabytes.  I'm thinking about 2-4TB of RAID.  Unless of course, RAID is
>>> obsolete these days.  Honestly, I find some of the levels of RAID
>>> confusing.  I want something that will survive a disk
>>> > >> failure (or two) out of the array.  Have any ideas, or can you
>>> point me to some place that discusses this somewhat intelligently?
>>> > >>
>>> > >> Are there reasonable systems that one can put together oneself
>>> these days?  Can I repurpose an older PC for this purpose?  Or an RPI4?
>>> What are the gotchas of going this way?
>>> > >>
>>> > >> I want to be able to set up a daily rsync or equivalent so we will
>>> lose as little as possible.  At the moment, I'm not thinking about
>>> surviving fire or disaster.  Maybe I should, but I suspect the costs
>>> balloon considerably.  I do not want to backup to the cloud because, plain
>>> and simple, I don't trust it to be fully secure.
>>> >
>>> > --
>>> > Connect with me on the GNU social network! <
>>> https://status.hackerposse.com/rozzin>
>>> > Not on the network? Ask me for more info!
>>> > _______________________________________________
>>> > gnhlug-discuss mailing list
>>> > gnhlug-discuss at mail.gnhlug.org
>>> > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>>
>>> _______________________________________________
>>> gnhlug-discuss mailing list
>>> gnhlug-discuss at mail.gnhlug.org
>>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>>
>>
>>
>> --
>> Marc Nozell (marc at nozell.com) http://www.nozell.com/blog
>> _______________________________________________
>> gnhlug-discuss mailing list
>> gnhlug-discuss at mail.gnhlug.org
>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>
>
> _______________________________________________
> gnhlug-discuss mailing listgnhlug-discuss at mail.gnhlug.orghttp://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210526/3d4b4a69/attachment-0001.html 