Have suggestions for a "roll your own file server"?
Bruce Labitt
bruce.labitt at myfairpoint.net
Wed May 26 10:59:35 EDT 2021
Finally back to this. Built a stack of metal plates that house my RPI4,
a boot SSD, a 1TB RAID1 array, and both active and passive USB3 hubs.
Machined parts so everything is bolted and clamped down. Have a PWM fan
that cools the RPI4 proportional to load that runs under systemd.
System boots from SSD. (No SD card.) It's kind of a brick sh!thouse,
but it's sturdy. Have created the RAID1 device - or it will be finished
in 45 minutes. It is still syncing.
Now I'd like to add the md0 device to /etc/fstab. The example I see is
with the device name. From https://www.tecmint.com/create-raid1-in-linux/
/dev/md0 /mnt/raid1 ext4 defaults 0 0
I've read it is better to use the UUID. Is the following the correct
syntax?
PARTUUID=my_complete_md0_UUID /mnt/raid1 ext4 defaults 0 0
where my_complete_md0_UUID comes from
$ lsblk -o UUID /dev/md0
Does one need to use tabs in fstab, or are spaces ok?
Once I figure this out - I have to figure out some rsync magic. Is it
better for the server to initiate the rsync, or the remote devices?
After all this I have to make another one. That shouldn't take as long
as the first time! For some pictures of the hardware build see
https://www.hobby-machinist.com/threads/an-rpi4-based-file-server.92273/#post-846939
On 3/10/21 8:49 PM, Bruce Labitt wrote:
> I'll take a look at that. Thanks for the link.
>
> On Wed, Mar 10, 2021 at 8:15 PM Marc Nozell (marc at nozell.com
> <mailto:marc at nozell.com>) <nozell at gmail.com <mailto:nozell at gmail.com>>
> wrote:
>
> Just to put a plug in for a colleague's work:
> https://perfectmediaserver.com/ <https://perfectmediaserver.com/>
> It covers everything from disk purchasing strategies, burn-in,
> filesystems (ZFS, SnapRAID, etc).
>
> He also hosts a podcast that folks here may find interesting:
> https://selfhosted.show/ <https://selfhosted.show/>
>
> -marc
>
> On Wed, Mar 10, 2021 at 8:08 PM <jonhall80 at comcast.net
> <mailto:jonhall80 at comcast.net>> wrote:
>
> OK:
>
> s/RPi4/some-other-cheap-computer-with-USB-3.x>/g
>
> Unless you build multiple Ethernet or WiFi or LTE modem
> connections your networking will still be the slowest thing.
>
> You do not need huge amounts of CPU power, or huge amounts of RAM.
>
> My basic point is that if you stick with simple RAID (like
> mirroring) but also set up a unit that is remote from your own
> home you could protect your own data from fire, flood and
> theft to a reasonable level and even protect your friend's
> data by backing up their data to your device.
>
> Add snapshots as suggested by Tom Buskey,perhaps encryption of
> file systems and data-streams and you can have a rather
> simple, server where you learn a lot by planning it out and
> setting it up rather than buying an "off the shelf" solution
> or simply using a "web backup".
>
> And good catch on the USB power supply.
>
> md
> > On 03/10/2021 6:53 PM Joshua Judson Rosen
> <rozzin at hackerposse.com <mailto:rozzin at hackerposse.com>> wrote:
> >
> >
> > I'm not sure about the Raspberry Pi 4, but up thru the raspi
> 3+ there are... problems, e.g.:
> >
> > Beware of USB on the raspi: there are some bugs in the
> silicon that pretty severely
> > cripple performance when multiple `bulk' devices are used at
> simultaneously,
> > sometimes to the point of making it unusable (e.g. if you
> want to use a better Wi-Fi
> > adapter/antenna than the one built onto the board, and
> connect an LTE modem so that
> > your raspi roam onto that if Wi-Fi becomes unavailable,
> throughput on whichever of those
> > interfaces you're actually using can become abysmal). IIRC
> the issue is basically
> > that the number of USB endpoints that can be assigned
> interrupts by the raspi controller
> > is _incredibly small_; and it's common for high-throughput
> devices to have multiple endpoints per device--
> > sometimes even one USB device will have more endpoints that
> the raspi USB controller can handle.
> >
> > Also, `network fileserver with USB-attached hard drives' is
> kind of the `peak unfitness'
> > for the raspberry pi. Specifically if you've got it attached
> to ethernet,
> > the ethernet is attached through the same slow-ish USB bus
> as your HDDs.
> >
> > (the onboard Wi-Fi BTW is SDIO; so if you avoid using the
> onboard Wi-Fi, I guess you might also
> > be able to make your µSD card faster...)
> >
> > ALSO: you'll really want to use an externally-powered USB
> hub for USB devices
> > that are not totally trivial, because the raspi's µUSB power
> supply is already
> > strained... (and if you're trying to power your raspi from
> some random USB power supply,
> > don't. Ideally you power it through the 5V pins on the
> expansion header...).
> >
> >
> > Though there is a lot of neat stuff that can be done with a
> Raspberry Pi,
> > it's really easy to overestimate it.
> >
> > But on the other hand: YMMV, and there are scenarios where
> the issues don't matter,
> > and might not even be noticeable. e.g., if you're dumping
> periodic backups to your
> > raspi asynchronously instead of (say) NFS mounting it and
> trying to use it interactively,
> > you might not even notice the weird bottlenecks because
> you're never looking at them.
> > And if you have enough of them as spares running
> simultaneously, you may not care
> > that every once in a while your fileystems get corrupted or
> your USB ports stop working
> > or whatever.
> >
> >
> > On 3/8/21 9:56 PM, jonhall80 at comcast.net
> <mailto:jonhall80 at comcast.net> wrote:
> > > I will suggest something and let people rip it apart:
> > >
> > > Get two RPis that have at least USB 2.0 Attach two large
> capacity disks to each one in a RAID-1 configuration (also
> known as "mirroring") to keep it simple. If one disk fails
> the other will still keep working (but you should replace it
> as soon as possible).
> > >
> > > Put all of your data on both systems.
> > >
> > > Take one of your systems to a friends or relatives house
> who you trust that has relatively good WiFi. Make sure the
> friend is relatively close, but is not in the same flood plain
> or fire area you are.
> > >
> > > Do an rsync every night to keep them in sync.
> > >
> > > Help your friend/relative do the same thing, keeping a
> copy of their data in your house. If your disks are big
> enough you could share systems and disks.
> > >
> > > Use encryption as you wish.
> > >
> > > Disk failure? Replace the disk and the data will be
> replicated.
> > > Fire, theft, earthquake? Take the replaced system over
> to your friends/relatives and copy the data at high speed,
> then take the copied system back to your house and start using
> it again.
> > >
> > > You would need three disks to fail at relatively the same
> time to lose your data. Or an asteroid crashing that wipes
> out all life on the planet. Unlikely.
> > >
> > > Realize that nothing is forever.
> > >
> > > md
> > >> On 03/08/2021 7:33 PM Bruce Labitt <bdlabitt at gmail.com
> <mailto:bdlabitt at gmail.com>> wrote:
> > >>
> > >>
> > >> For the second time in 3 months I have had a computer
> failure. Oddly, it was a PS on the motherboard both times.
> (Two different MB's.) Fortunately the disks were ok. I'm
> living on borrowed time. Next time, I may not be that lucky.
> > >>
> > >> Need a file server system with some sort of RAID
> redundancy. I want to backup 2 main computers, plus photos.
> Maybe this RPI4 too, since that's what I'm running on, due to
> the second failure. If this SSD goes, I'm gonna be a
> sad puppy. This is for home use, so we are not talking
> Exabytes. I'm thinking about 2-4TB of RAID. Unless of course,
> RAID is obsolete these days. Honestly, I find some of the
> levels of RAID confusing. I want something that will survive
> a disk
> > >> failure (or two) out of the array. Have any ideas, or
> can you point me to some place that discusses this somewhat
> intelligently?
> > >>
> > >> Are there reasonable systems that one can put together
> oneself these days? Can I repurpose an older PC for this
> purpose? Or an RPI4? What are the gotchas of going this way?
> > >>
> > >> I want to be able to set up a daily rsync or equivalent
> so we will lose as little as possible. At the moment, I'm not
> thinking about surviving fire or disaster. Maybe I should,
> but I suspect the costs balloon considerably. I do not want
> to backup to the cloud because, plain and simple, I don't
> trust it to be fully secure.
> >
> > --
> > Connect with me on the GNU social network!
> <https://status.hackerposse.com/rozzin
> <https://status.hackerposse.com/rozzin>>
> > Not on the network? Ask me for more info!
> > _______________________________________________
> > gnhlug-discuss mailing list
> > gnhlug-discuss at mail.gnhlug.org
> <mailto:gnhlug-discuss at mail.gnhlug.org>
> > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
> <http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/>
>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> <mailto:gnhlug-discuss at mail.gnhlug.org>
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
> <http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/>
>
>
>
> --
> Marc Nozell (marc at nozell.com <mailto:marc at nozell.com>)
> http://www.nozell.com/blog <http://www.nozell.com/blog>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org <mailto:gnhlug-discuss at mail.gnhlug.org>
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
> <http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/>
>
>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210526/ac5c68d5/attachment-0001.html
More information about the gnhlug-discuss
mailing list