Have suggestions for a "roll your own file server"?
Bruce Labitt
bdlabitt at gmail.com
Wed May 26 16:58:40 EDT 2021
Thanks for the compliment. Put a bit of work into it. Self taught hobby
machinist. Self taught Linux as well. Only have had machines for 18
months. Embarrassed to say how long I've been using Linux, as I keep on
asking basic questions.
On Wed, May 26, 2021, 12:14 PM Tom Buskey <tom at buskey.name> wrote:
> My Fedora /etc/fstab has spaces
> UUID=54103729-6e0a-4345-a2b8-8b8cded29ee1 /boot ext4
> defaults 1 2
>
> I've had clients initiate rsync for security. I think the client
> initiation would offload the rsync compute from the server.
> For a home server, it's nice to just monitor the server instead of
> multiple clients.
>
> Nice buiild
>
> On Wed, May 26, 2021 at 11:00 AM Bruce Labitt <
> bruce.labitt at myfairpoint.net> wrote:
>
>> Finally back to this. Built a stack of metal plates that house my RPI4,
>> a boot SSD, a 1TB RAID1 array, and both active and passive USB3 hubs.
>> Machined parts so everything is bolted and clamped down. Have a PWM fan
>> that cools the RPI4 proportional to load that runs under systemd. System
>> boots from SSD. (No SD card.) It's kind of a brick sh!thouse, but it's
>> sturdy. Have created the RAID1 device - or it will be finished in 45
>> minutes. It is still syncing.
>>
>> Now I'd like to add the md0 device to /etc/fstab. The example I see is
>> with the device name. From
>> https://www.tecmint.com/create-raid1-in-linux/
>> /dev/md0 /mnt/raid1 ext4 defaults 0 0
>>
>> I've read it is better to use the UUID. Is the following the correct
>> syntax?
>>
>> PARTUUID=my_complete_md0_UUID /mnt/raid1 ext4 defaults 0 0
>>
>> where my_complete_md0_UUID comes from
>> $ lsblk -o UUID /dev/md0
>>
>> Does one need to use tabs in fstab, or are spaces ok?
>>
>> Once I figure this out - I have to figure out some rsync magic. Is it
>> better for the server to initiate the rsync, or the remote devices?
>>
>> After all this I have to make another one. That shouldn't take as long
>> as the first time! For some pictures of the hardware build see
>> https://www.hobby-machinist.com/threads/an-rpi4-based-file-server.92273/#post-846939
>>
>>
>>
>> On 3/10/21 8:49 PM, Bruce Labitt wrote:
>>
>> I'll take a look at that. Thanks for the link.
>>
>> On Wed, Mar 10, 2021 at 8:15 PM Marc Nozell (marc at nozell.com) <
>> nozell at gmail.com> wrote:
>>
>>> Just to put a plug in for a colleague's work:
>>> https://perfectmediaserver.com/ It covers everything from disk
>>> purchasing strategies, burn-in, filesystems (ZFS, SnapRAID, etc).
>>>
>>> He also hosts a podcast that folks here may find interesting:
>>> https://selfhosted.show/
>>>
>>> -marc
>>>
>>> On Wed, Mar 10, 2021 at 8:08 PM <jonhall80 at comcast.net> wrote:
>>>
>>>> OK:
>>>>
>>>> s/RPi4/some-other-cheap-computer-with-USB-3.x>/g
>>>>
>>>> Unless you build multiple Ethernet or WiFi or LTE modem connections
>>>> your networking will still be the slowest thing.
>>>>
>>>> You do not need huge amounts of CPU power, or huge amounts of RAM.
>>>>
>>>> My basic point is that if you stick with simple RAID (like mirroring)
>>>> but also set up a unit that is remote from your own home you could protect
>>>> your own data from fire, flood and theft to a reasonable level and even
>>>> protect your friend's data by backing up their data to your device.
>>>>
>>>> Add snapshots as suggested by Tom Buskey,perhaps encryption of file
>>>> systems and data-streams and you can have a rather simple, server where you
>>>> learn a lot by planning it out and setting it up rather than buying an "off
>>>> the shelf" solution or simply using a "web backup".
>>>>
>>>> And good catch on the USB power supply.
>>>>
>>>> md
>>>> > On 03/10/2021 6:53 PM Joshua Judson Rosen <rozzin at hackerposse.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > I'm not sure about the Raspberry Pi 4, but up thru the raspi 3+ there
>>>> are... problems, e.g.:
>>>> >
>>>> > Beware of USB on the raspi: there are some bugs in the silicon that
>>>> pretty severely
>>>> > cripple performance when multiple `bulk' devices are used at
>>>> simultaneously,
>>>> > sometimes to the point of making it unusable (e.g. if you want to use
>>>> a better Wi-Fi
>>>> > adapter/antenna than the one built onto the board, and connect an LTE
>>>> modem so that
>>>> > your raspi roam onto that if Wi-Fi becomes unavailable, throughput on
>>>> whichever of those
>>>> > interfaces you're actually using can become abysmal). IIRC the issue
>>>> is basically
>>>> > that the number of USB endpoints that can be assigned interrupts by
>>>> the raspi controller
>>>> > is _incredibly small_; and it's common for high-throughput devices to
>>>> have multiple endpoints per device--
>>>> > sometimes even one USB device will have more endpoints that the raspi
>>>> USB controller can handle.
>>>> >
>>>> > Also, `network fileserver with USB-attached hard drives' is kind of
>>>> the `peak unfitness'
>>>> > for the raspberry pi. Specifically if you've got it attached to
>>>> ethernet,
>>>> > the ethernet is attached through the same slow-ish USB bus as your
>>>> HDDs.
>>>> >
>>>> > (the onboard Wi-Fi BTW is SDIO; so if you avoid using the onboard
>>>> Wi-Fi, I guess you might also
>>>> > be able to make your µSD card faster...)
>>>> >
>>>> > ALSO: you'll really want to use an externally-powered USB hub for USB
>>>> devices
>>>> > that are not totally trivial, because the raspi's µUSB power supply
>>>> is already
>>>> > strained... (and if you're trying to power your raspi from some
>>>> random USB power supply,
>>>> > don't. Ideally you power it through the 5V pins on the expansion
>>>> header...).
>>>> >
>>>> >
>>>> > Though there is a lot of neat stuff that can be done with a Raspberry
>>>> Pi,
>>>> > it's really easy to overestimate it.
>>>> >
>>>> > But on the other hand: YMMV, and there are scenarios where the issues
>>>> don't matter,
>>>> > and might not even be noticeable. e.g., if you're dumping periodic
>>>> backups to your
>>>> > raspi asynchronously instead of (say) NFS mounting it and trying to
>>>> use it interactively,
>>>> > you might not even notice the weird bottlenecks because you're never
>>>> looking at them.
>>>> > And if you have enough of them as spares running simultaneously, you
>>>> may not care
>>>> > that every once in a while your fileystems get corrupted or your USB
>>>> ports stop working
>>>> > or whatever.
>>>> >
>>>> >
>>>> > On 3/8/21 9:56 PM, jonhall80 at comcast.net wrote:
>>>> > > I will suggest something and let people rip it apart:
>>>> > >
>>>> > > Get two RPis that have at least USB 2.0 Attach two large capacity
>>>> disks to each one in a RAID-1 configuration (also known as "mirroring") to
>>>> keep it simple. If one disk fails the other will still keep working (but
>>>> you should replace it as soon as possible).
>>>> > >
>>>> > > Put all of your data on both systems.
>>>> > >
>>>> > > Take one of your systems to a friends or relatives house who you
>>>> trust that has relatively good WiFi. Make sure the friend is relatively
>>>> close, but is not in the same flood plain or fire area you are.
>>>> > >
>>>> > > Do an rsync every night to keep them in sync.
>>>> > >
>>>> > > Help your friend/relative do the same thing, keeping a copy of
>>>> their data in your house. If your disks are big enough you could share
>>>> systems and disks.
>>>> > >
>>>> > > Use encryption as you wish.
>>>> > >
>>>> > > Disk failure? Replace the disk and the data will be replicated.
>>>> > > Fire, theft, earthquake? Take the replaced system over to your
>>>> friends/relatives and copy the data at high speed, then take the copied
>>>> system back to your house and start using it again.
>>>> > >
>>>> > > You would need three disks to fail at relatively the same time to
>>>> lose your data. Or an asteroid crashing that wipes out all life on the
>>>> planet. Unlikely.
>>>> > >
>>>> > > Realize that nothing is forever.
>>>> > >
>>>> > > md
>>>> > >> On 03/08/2021 7:33 PM Bruce Labitt <bdlabitt at gmail.com> wrote:
>>>> > >>
>>>> > >>
>>>> > >> For the second time in 3 months I have had a computer failure.
>>>> Oddly, it was a PS on the motherboard both times. (Two different MB's.)
>>>> Fortunately the disks were ok. I'm living on borrowed time. Next time, I
>>>> may not be that lucky.
>>>> > >>
>>>> > >> Need a file server system with some sort of RAID redundancy.
>>>> I want to backup 2 main computers, plus photos. Maybe this RPI4 too, since
>>>> that's what I'm running on, due to the second failure. If this SSD goes,
>>>> I'm gonna be a sad puppy. This is for home use, so we are not talking
>>>> Exabytes. I'm thinking about 2-4TB of RAID. Unless of course, RAID is
>>>> obsolete these days. Honestly, I find some of the levels of RAID
>>>> confusing. I want something that will survive a disk
>>>> > >> failure (or two) out of the array. Have any ideas, or can you
>>>> point me to some place that discusses this somewhat intelligently?
>>>> > >>
>>>> > >> Are there reasonable systems that one can put together oneself
>>>> these days? Can I repurpose an older PC for this purpose? Or an RPI4?
>>>> What are the gotchas of going this way?
>>>> > >>
>>>> > >> I want to be able to set up a daily rsync or equivalent so we will
>>>> lose as little as possible. At the moment, I'm not thinking about
>>>> surviving fire or disaster. Maybe I should, but I suspect the costs
>>>> balloon considerably. I do not want to backup to the cloud because, plain
>>>> and simple, I don't trust it to be fully secure.
>>>> >
>>>> > --
>>>> > Connect with me on the GNU social network! <
>>>> https://status.hackerposse.com/rozzin>
>>>> > Not on the network? Ask me for more info!
>>>> > _______________________________________________
>>>> > gnhlug-discuss mailing list
>>>> > gnhlug-discuss at mail.gnhlug.org
>>>> > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>>>
>>>> _______________________________________________
>>>> gnhlug-discuss mailing list
>>>> gnhlug-discuss at mail.gnhlug.org
>>>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>>>
>>>
>>>
>>> --
>>> Marc Nozell (marc at nozell.com) http://www.nozell.com/blog
>>> _______________________________________________
>>> gnhlug-discuss mailing list
>>> gnhlug-discuss at mail.gnhlug.org
>>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>>
>>
>> _______________________________________________
>> gnhlug-discuss mailing listgnhlug-discuss at mail.gnhlug.orghttp://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>
>>
>> _______________________________________________
>> gnhlug-discuss mailing list
>> gnhlug-discuss at mail.gnhlug.org
>> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210526/34bd8fe3/attachment-0001.html
More information about the gnhlug-discuss
mailing list