rsync question

jonhall80 at comcast.net jonhall80 at comcast.net
Tue Mar 9 23:13:08 EST 2021


Yes, cp -a is a more efficient method of copying a complete file system to another file system that is empty.   But if the mirrored file system is only being updated then rsync is much more efficient, both for the reasons that Ken stated and because if the files are the same (as determined by file size and modification time) then the copy is not done.

Another thing that could speed the copy up is to mount the file system as one that does not use a log file.   The log of a file system is used when the system crashes.  When the system crashes the buffer that holds the data and metadata of the file system would disappear, and the file system would be corrupted.  This is why an fsck() had to be done in the old days.   You can control the creation and use of the log file when you use ext3 or ext4.   ext2 does not use a log file at all.  If you are just making a "backup" of the file system you might as well use ext2 IMHO.

Log-based systems write the metadata to the log first so if the system crashes when the system reboots the metadata is applied and the file system is complete.   Various other file systems may also write the data to the log file first due to the features of the file system layout.

When a sync is done all of the data and metadata is applied to the file system and the log is "rewound" effectively erasing it and starting over again.   The same thing happens if the file system is unmounted or if the system is shut down gracefully.

However all of this log activity generates more reads, writes and data transfer, which slows the copying down.

If you are copying one disk to another you are in control.   If the copy completes, you unmount the new filesystem and all of the data is written to the disk.  The log was never used and all that writing (and data throughput) was for naught.

If the system crashes, when it comes back up you simply recreate the filesystem and start the copy over again.

99.9999% of the time the copy will complete but you will save all of that writing and re-writing.

This is particularly visible on slow USB devices.

Everything other than USB 4 is slow.

YMMV

md




>     On 03/09/2021 9:54 PM Ken D'Ambrosio <ken at jots.org> wrote:
>      
>      
> 
>     On 2021-03-09 21:29, Bruce Labitt wrote:
> 
>         > >         A maybe not so smart rsync question...
> > 
> >         If one uses rsync -avz src/bar  /disk2    will that copy over everything from src/bar and create a directory bar on disk2?  What if src/bar has other users or root?  In other words, does the -a mean that it will preserve ownership and links and copy to /disk2?  Just don't know if I need sudo or not.
> >          
> >         I dumbly did a copy.  Well, that didn't preserve permissions or attributes.  So deleting that...  Since I'm trying to back up 100's of GB, thought I'd ask.  This is taking a long time, even with USB3 drives and nvme.  
> > 
> >     > 
>     The '-a' flag is one of the few flags that actually means the same thing for both rsync and cp -- and does much the same thing, too.  You should be able to use cp -av /src/bar /disk2 and have it copy attributes/permissions.  You need the "sudo" in there to preserve ownership, as non-root can't assign other users' ownerships to the files.  The main reason I might recommend "cp" over "rsync" is simply because I find "cp" a little less confusing -- what "rsync" does can vary depending on whether or not there are trailing slashes and so forth, and I never quite remember which is which.  But the best thing about rsync is that if it barfs in the middle, you just up-arrow, and it starts over from where you left off.  As with "cp", if you want the ownerships preserved, don't forget the "sudo".  Note that unless you're transiting a network, I'd leave the "-z" off, because it'll just slow you down (substantially).
> 
>     -Ken
> 
>     _______________________________________________
>     gnhlug-discuss mailing list
>     gnhlug-discuss at mail.gnhlug.org
>     http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.gnhlug.org/pipermail/gnhlug-discuss/attachments/20210309/25ad0caa/attachment.html 


More information about the gnhlug-discuss mailing list