Have suggestions for a "roll your own file server"?

Wed Mar 10 20:14:31 EST 2021

Just to put a plug in for a colleague's work:
https://perfectmediaserver.com/    It covers everything from disk
purchasing strategies, burn-in, filesystems (ZFS, SnapRAID, etc).

He also hosts a podcast that folks here may find interesting:
https://selfhosted.show/

-marc

On Wed, Mar 10, 2021 at 8:08 PM <jonhall80 at comcast.net> wrote:

> OK:
>
> s/RPi4/some-other-cheap-computer-with-USB-3.x>/g
>
> Unless you build multiple Ethernet or WiFi or LTE modem connections your
> networking will still be the slowest thing.
>
> You do not need huge amounts of CPU power, or huge amounts of RAM.
>
> My basic point is that if you stick with simple RAID (like mirroring) but
> also set up a unit that is remote from your own home you could protect your
> own data from fire, flood and theft to a reasonable level and even protect
> your friend's data by backing up their data to your device.
>
> Add snapshots as suggested by Tom Buskey,perhaps encryption of file
> systems and data-streams and you can have a rather simple, server where you
> learn a lot by planning it out and setting it up rather than buying an "off
> the shelf" solution or simply using a "web backup".
>
> And good catch on the USB power supply.
>
> md
> > On 03/10/2021 6:53 PM Joshua Judson Rosen <rozzin at hackerposse.com>
> wrote:
> >
> >
> > I'm not sure about the Raspberry Pi 4, but up thru the raspi 3+ there
> are... problems, e.g.:
> >
> > Beware of USB on the raspi: there are some bugs in the silicon that
> pretty severely
> > cripple performance when multiple `bulk' devices are used at
> simultaneously,
> > sometimes to the point of making it unusable (e.g. if you want to use a
> better Wi-Fi
> > adapter/antenna than the one built onto the board, and connect an LTE
> modem so that
> > your raspi roam onto that if Wi-Fi becomes unavailable, throughput on
> whichever of those
> > interfaces you're actually using can become abysmal). IIRC the issue is
> basically
> > that the number of USB endpoints that can be assigned interrupts by the
> raspi controller
> > is _incredibly small_; and it's common for high-throughput devices to
> have multiple endpoints per device--
> > sometimes even one USB device will have more endpoints that the raspi
> USB controller can handle.
> >
> > Also, `network fileserver with USB-attached hard drives' is kind of the
> `peak unfitness'
> > for the raspberry pi. Specifically if you've got it attached to ethernet,
> > the ethernet is attached through the same slow-ish USB bus as your HDDs.
> >
> > (the onboard Wi-Fi BTW is SDIO; so if you avoid using the onboard Wi-Fi,
> I guess you might also
> >  be able to make your µSD card faster...)
> >
> > ALSO: you'll really want to use an externally-powered USB hub for USB
> devices
> > that are not totally trivial, because the raspi's µUSB power supply is
> already
> > strained... (and if you're trying to power your raspi from some random
> USB power supply,
> > don't. Ideally you power it through the 5V pins on the expansion
> header...).
> >
> >
> > Though there is a lot of neat stuff that can be done with a Raspberry Pi,
> > it's really easy to overestimate it.
> >
> > But on the other hand: YMMV, and there are scenarios where the issues
> don't matter,
> > and might not even be noticeable. e.g., if you're dumping periodic
> backups to your
> > raspi asynchronously instead of (say) NFS mounting it and trying to use
> it interactively,
> > you might not even notice the weird bottlenecks because you're never
> looking at them.
> > And if you have enough of them as spares running simultaneously, you may
> not care
> > that every once in a while your fileystems get corrupted or your USB
> ports stop working
> > or whatever.
> >
> >
> > On 3/8/21 9:56 PM, jonhall80 at comcast.net wrote:
> > > I will suggest something and let people rip it apart:
> > >
> > > Get two RPis that have at least USB 2.0  Attach two large capacity
> disks to each one in a RAID-1 configuration (also known as "mirroring") to
> keep it simple.  If one disk fails the other will still keep working (but
> you should replace it as soon as possible).
> > >
> > > Put all of your data on both systems.
> > >
> > > Take one of your systems to a friends or relatives house who you trust
> that has relatively good WiFi.  Make sure the friend is relatively close,
> but is not in the same flood plain or fire area you are.
> > >
> > > Do an rsync every night to keep them in sync.
> > >
> > > Help your friend/relative do the same thing, keeping a copy of their
> data in your house.   If your disks are big enough you could share systems
> and disks.
> > >
> > > Use encryption as you wish.
> > >
> > > Disk failure?   Replace the disk and the data will be replicated.
> > > Fire, theft, earthquake?   Take the replaced system over to your
> friends/relatives and copy the data at high speed, then take the copied
> system back to your house and start using it again.
> > >
> > > You would need three disks to fail at relatively the same time to lose
> your data.   Or an asteroid crashing that wipes out all life on the
> planet.  Unlikely.
> > >
> > > Realize that nothing is forever.
> > >
> > > md
> > >> On 03/08/2021 7:33 PM Bruce Labitt <bdlabitt at gmail.com> wrote:
> > >>
> > >>
> > >> For the second time in 3 months I have had a computer failure.
> Oddly, it was a PS on the motherboard both times.  (Two different MB's.)
> Fortunately the disks were ok.  I'm living on borrowed time.  Next time, I
> may not be that lucky.
> > >>
> > >> Need a file server system with some sort of RAID redundancy.  I want
> to backup 2 main computers, plus photos.  Maybe this RPI4 too, since that's
> what I'm running on, due to the second failure.  If this SSD goes, I'm
> gonna be a sad puppy.  This is for home use, so we are not talking
> Exabytes.  I'm thinking about 2-4TB of RAID.  Unless of course, RAID is
> obsolete these days.  Honestly, I find some of the levels of RAID
> confusing.  I want something that will survive a disk
> > >> failure (or two) out of the array.  Have any ideas, or can you point
> me to some place that discusses this somewhat intelligently?
> > >>
> > >> Are there reasonable systems that one can put together oneself these
> days?  Can I repurpose an older PC for this purpose?  Or an RPI4?  What are
> the gotchas of going this way?
> > >>
> > >> I want to be able to set up a daily rsync or equivalent so we will
> lose as little as possible.  At the moment, I'm not thinking about
> surviving fire or disaster.  Maybe I should, but I suspect the costs
> balloon considerably.  I do not want to backup to the cloud because, plain
> and simple, I don't trust it to be fully secure.
> >
> > --
> > Connect with me on the GNU social network! <
> https://status.hackerposse.com/rozzin>
> > Not on the network? Ask me for more info!
> > _______________________________________________
> > gnhlug-discuss mailing list
> > gnhlug-discuss at mail.gnhlug.org
> > http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>

-- 
Marc Nozell (marc at nozell.com) http://www.nozell.com/blog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210310/7ca28e5f/attachment.html 