rsync - Aug 2013 - [Bug 10074] New: rsync reorders --from-files alphabetically

https://bugzilla.samba.org/show_bug.cgi?id=10074

           Summary: rsync reorders --from-files alphabetically
           Product: rsync
           Version: 3.0.6
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: nicholas.man at gmail.com
         QAContact: rsync-qa at samba.org


This is request to add a flag to request that rsync does not automatically sort
--from-files alphabetically, due to its use case with LTFS media.

LTFS is a linear media with exceedingly slow seek times when compared with raw
read/write speed. To do efficient reads, each file on the tape must be read off
in the order it was written. Each file has an extended attribute
(ltfs.startblock) which returns its exact location (startblock) on the tape. In
this manner, you can easily obtain a list of files in the optimal order for
reads (sorted, ascending, in order of their startblock).

rsync's behavior of copying all files on and off in alphabetical order works
fine if all you ever do is one write, as the order is consistent. If you do
anything else to that volume, however, (using, for example, rsync -u to copy
additional files), it will copy them on in alphabetical order but multiple
times. You will end up with n number of a-z writes distributed across 6 minutes
of seek time.

Trying to restore alphabetically after doing more than one copy will result in
one of the least optimal restore paths imaginable, with average read rates
dropping by 50% in my case when compared with copying them one at a time using
rsync. I, however, was copying fairly large files. Any number of smaller sizes
or wider distribution of files, or both will decimate the performance. You
could easily create a file structure for which rsync would blindly shoot back
and fourth across the entire tape for every single file.

Basically: batch file copying with rsync, if it always restores alphabetically,
is useless for LTFS volumes.

While I can write a python script to execute rsync on individual files, it
would be nice to be able to pass an argument to rsync (--maintain-file-order)
which would prevent rsync from taking my optimally ordered list and butchering
the LTFS restore with an alphabetical sort.

--Nicholas Andre

(Reposted from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=640492)

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10074

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
            Version|3.0.6                       |3.1.0

--- Comment #1 from Wayne Davison <wayned at samba.org> 2013-08-12
05:57:59 UTC ---
It doesn't seem very hard to get something going that may meet your needs,
but
I don't think that I will add this as an official feature.  I'll attach
a patch
for 3.1.0 pre1.  To use it, you'll need to use --no-i-r (to avoid
incremental-recursion reordering) and possibly --no-i-d (if you need to avoid
implied directories).

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10074

--- Comment #2 from Wayne Davison <wayned at samba.org> 2013-08-12
06:00:04 UTC ---
Created attachment 9124
  --> https://bugzilla.samba.org/attachment.cgi?id=9124
Add the --unsorted option

This patch adds the --unsorted option, slightly extending the
need_unsorted_flist flag variable that is used by the iconv code.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10074

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #9124|0                           |1
        is obsolete|                            |

--- Comment #3 from Wayne Davison <wayned at samba.org> 2013-08-13
16:27:05 UTC ---
Created attachment 9130
  --> https://bugzilla.samba.org/attachment.cgi?id=9130
Add the --unsorted option

This version of the patch also insures that a remote rsync gets told about the
--unsorted option.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10074

--- Comment #4 from Nicholas Andre <nicholas.man at gmail.com> 2013-08-14
20:59:17 UTC ---
(In reply to comment #3)
> Created attachment 9130 [details]
> Add the --unsorted option
> 
> This version of the patch also insures that a remote rsync gets told about
the
> --unsorted option.
Tested, works great! Thanks!

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10074

--- Comment #5 from Christian Campbell <dcamp at alumni.ufl.edu>
2013-09-17 04:56:39 UTC ---
Thank you, Wayne -- works great for me, too! I was delighted to find that a
solution to my problem had so recently come into existence.  =-)

I'm curious why you don't think --unsorted should be an official
feature. It's
obviously critical for the OP's slow linear media problem, but it seems it
would eliminate a great amount of time lost to needlessly wide-angle random
seek distances even when reading from HDDs. (That's my use case, though
I'm not
profiling to prove the hypothesis.)

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10074

Dave Rice <dave at dericed.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dave at dericed.com

--- Comment #6 from Dave Rice <dave at dericed.com> ---
I agree with Christian Campbell this patch would be helpful as an official part
of rsync and encourage that this be considered as there's more and more
rsync
users working with LTFS.
Dave Rice

-- 
You are receiving this mail because:
You are the QA Contact for the bug.