Have suggestions for a "roll your own file server"?

[Bruce Labitt]

Actually, at the moment I am forced to run on a RPI4-4GB.  My laptop PS
died.  I'm aware of many of the downfalls of an RPI, I've had a few PI's
over the ages.  Mercifully, they are getting better.  One cannot run an SSD
AND and active USB3 hub both plugged directly into the 2 available USB3
ports.  The RPI4 will not boot.  The active hub's voltage backfeeds into
the Pi and the PI senses this, and prevents booting.  If however, the
active hub is buffered by a passive hub, it will work.  I discovered this
accidentally.

Anyhow, it's not too bad with two disks plugged into the active USB3 hub.
I had to rsync my laptop NVME to another NVME and it was sort of slow, but
I was transferring over 110GB.  Average transfer rate was over 90MB/sec for
all sorts of large and millions of small files.  Yeah not super fast, but
tolerable.  The file transfer was happening all the while I was using the
RPI4 for browsing with 14 active tabs.

I'm not using the SD card at the moment - that's unbearably slow.  I boot
from SSD.  (I lasted about a day using the RPI4 as a computer on an SD.  I
then ordered a $20 120GB SSD!)  Doing that, the RPI4 is tolerable,
especially with a slight overclock.  The RPI4 is a limited platform, but
it's not horrible, like it's predecessors were.  The earlier versions are
ok for print servers, sprinkler controllers and DNS filters (pihole).  I
think an RPI4 could make an ok host for a mirror array.  Don't know how to
do it yet, but it seems very approachable.

What I haven't figured out is how to have a remote offsite PI as well.
Just don't know how to secure it and be able to access it remotely.  At the
moment it's beyond my skill set.


On Wed, Mar 10, 2021 at 6:54 PM Joshua Judson Rosen <rozzin at hackerposse.com>
wrote:

> I'm not sure about the Raspberry Pi 4, but up thru the raspi 3+ there
> are... problems, e.g.:
>
> Beware of USB on the raspi: there are some bugs in the silicon that pretty
> severely
> cripple performance when multiple `bulk' devices are used at
> simultaneously,
> sometimes to the point of making it unusable (e.g. if you want to use a
> better Wi-Fi
> adapter/antenna than the one built onto the board, and connect an LTE
> modem so that
> your raspi roam onto that if Wi-Fi becomes unavailable, throughput on
> whichever of those
> interfaces you're actually using can become abysmal). IIRC the issue is
> basically
> that the number of USB endpoints that can be assigned interrupts by the
> raspi controller
> is _incredibly small_; and it's common for high-throughput devices to have
> multiple endpoints per device--
> sometimes even one USB device will have more endpoints that the raspi USB
> controller can handle.
>
> Also, `network fileserver with USB-attached hard drives' is kind of the
> `peak unfitness'
> for the raspberry pi. Specifically if you've got it attached to ethernet,
> the ethernet is attached through the same slow-ish USB bus as your HDDs.
>
> (the onboard Wi-Fi BTW is SDIO; so if you avoid using the onboard Wi-Fi, I
> guess you might also
>  be able to make your µSD card faster...)
>
> ALSO: you'll really want to use an externally-powered USB hub for USB
> devices
> that are not totally trivial, because the raspi's µUSB power supply is
> already
> strained... (and if you're trying to power your raspi from some random USB
> power supply,
> don't. Ideally you power it through the 5V pins on the expansion
> header...).
>
>
> Though there is a lot of neat stuff that can be done with a Raspberry Pi,
> it's really easy to overestimate it.
>
> But on the other hand: YMMV, and there are scenarios where the issues
> don't matter,
> and might not even be noticeable. e.g., if you're dumping periodic backups
> to your
> raspi asynchronously instead of (say) NFS mounting it and trying to use it
> interactively,
> you might not even notice the weird bottlenecks because you're never
> looking at them.
> And if you have enough of them as spares running simultaneously, you may
> not care
> that every once in a while your fileystems get corrupted or your USB ports
> stop working
> or whatever.
>
>
> On 3/8/21 9:56 PM, jonhall80 at comcast.net wrote:
> > I will suggest something and let people rip it apart:
> >
> > Get two RPis that have at least USB 2.0  Attach two large capacity disks
> to each one in a RAID-1 configuration (also known as "mirroring") to keep
> it simple.  If one disk fails the other will still keep working (but you
> should replace it as soon as possible).
> >
> > Put all of your data on both systems.
> >
> > Take one of your systems to a friends or relatives house who you trust
> that has relatively good WiFi.  Make sure the friend is relatively close,
> but is not in the same flood plain or fire area you are.
> >
> > Do an rsync every night to keep them in sync.
> >
> > Help your friend/relative do the same thing, keeping a copy of their
> data in your house.   If your disks are big enough you could share systems
> and disks.
> >
> > Use encryption as you wish.
> >
> > Disk failure?   Replace the disk and the data will be replicated.
> > Fire, theft, earthquake?   Take the replaced system over to your
> friends/relatives and copy the data at high speed, then take the copied
> system back to your house and start using it again.
> >
> > You would need three disks to fail at relatively the same time to lose
> your data.   Or an asteroid crashing that wipes out all life on the
> planet.  Unlikely.
> >
> > Realize that nothing is forever.
> >
> > md
> >> On 03/08/2021 7:33 PM Bruce Labitt <bdlabitt at gmail.com> wrote:
> >>
> >>
> >> For the second time in 3 months I have had a computer failure.  Oddly,
> it was a PS on the motherboard both times.  (Two different MB's.)
> Fortunately the disks were ok.  I'm living on borrowed time.  Next time, I
> may not be that lucky.
> >>
> >> Need a file server system with some sort of RAID redundancy.  I want to
> backup 2 main computers, plus photos.  Maybe this RPI4 too, since that's
> what I'm running on, due to the second failure.  If this SSD goes, I'm
> gonna be a sad puppy.  This is for home use, so we are not talking
> Exabytes.  I'm thinking about 2-4TB of RAID.  Unless of course, RAID is
> obsolete these days.  Honestly, I find some of the levels of RAID
> confusing.  I want something that will survive a disk
> >> failure (or two) out of the array.  Have any ideas, or can you point me
> to some place that discusses this somewhat intelligently?
> >>
> >> Are there reasonable systems that one can put together oneself these
> days?  Can I repurpose an older PC for this purpose?  Or an RPI4?  What are
> the gotchas of going this way?
> >>
> >> I want to be able to set up a daily rsync or equivalent so we will lose
> as little as possible.  At the moment, I'm not thinking about surviving
> fire or disaster.  Maybe I should, but I suspect the costs balloon
> considerably.  I do not want to backup to the cloud because, plain and
> simple, I don't trust it to be fully secure.
>
> --
> Connect with me on the GNU social network! <
> https://status.hackerposse.com/rozzin>
> Not on the network? Ask me for more info!
> _______________________________________________
> gnhlug-discuss mailing list
> gnhlug-discuss at mail.gnhlug.org
> http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210310/846422bb/attachment-0001.html