Brian "bex" Exelbierd
2023-Sep-12 09:22 UTC
Feature Concept: enable iCloud Drive with rsync
Hi, I have also posted this on GitHub but it isn?t clear that was the right place: https://github.com/WayneD/rsync/issues/522 iCloud Drive will evict files that are unused or when additional space is needed on the local drive. The evicted files are replace by "bookmark" files that allow MacOS to continue to report the files in the file system as though they were actually present. The files are downloaded again either on request or when needed. rsync, like all similar tools I can find, doesn't have any way of handling these evicted files. I have been thinking about this and I think I know how to make it work. The short explanation is that iCloud Drive preserves access to the required metadata. Here is an abbreviated output from `mdls` of an evicted file: ``` % mdls .1998-tax-return.pdf.icloud kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000 kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000 kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000 kMDItemDateAdded = 2021-11-05 15:42:15 +0000 kMDItemDisplayName = "1998-tax-return.pdf" kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000 kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000 kMDItemFSSize = 1929932 kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000 kMDItemLogicalSize = 1929932 kMDItemPhysicalSize = 1929932 ``` This metadata, I think, is enough to pass the rsync quick check as described in the man page. Therefore, I suspect what needs to happen, from a code perspective, is that rsync needs to be modified to do the following when it finds an evicted file. All evicted files are named consistently, `.<filename>.iCloud` so they are easy to spot. 1. Perform the system call equivalent of the mdls above to obtain the appropriate dates and sizes. 2. If the file cannot be skipped because it either has changed or a checksum is required, perform the system call equivalent of `brctl download <filename>` to get the file downloaded. 3. Rsync the file 4. If the file was downloaded by rsync, perform the system call equivalent of `brctl evict <filename>` to remove the file to leave the system in the same state. This simplistic algorithm would leave some open issues/caveats: - It is likely that the rsync can only be run one-way from iCloud Drive to non-iCloud Drive data storage. While it is possible it could be run two-way research is needed on whether you?d have to download the old file before you replace it. - Timeouts may happen. iCloud Drive still gets stuck sometimes and there will be non-zero time during downloads that don?t get stuck. - There is no guarantee there is ever enough room on disk to hold a specific file from iCloud Drive. This is likely resolved by MacOS directly, however, this may take excessive time. - If you thought running with checksums was slow before, strap in, because those downloads are going to add up. All that said, I think this would accomplish a high percentage of what users are looking for. Is this kind of a patch welcome? Is the plan sound? I haven't written C since college (in the 90s!) and would be open to advice/mentoring. Thank you. regards, bex -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20230912/5dc50a2c/attachment.htm>
Is this being accessed via a fuse mount? If so it seems like that is where this kind of feature should be implemented (like a mount option to decide how to handle such files). Rsync shouldn't need special features to deal with every kind of file storage. On 9/12/23 05:22, Brian "bex" Exelbierd via rsync wrote:> Hi, > > I have also posted this on GitHub but it isn?t clear that was the right > place: https://github.com/WayneD/rsync/issues/522 > > iCloud Drive will evict files that are unused or when additional space > is needed on the local drive. The evicted files are replace by > "bookmark" files that allow MacOS to continue to report the files in the > file system as though they were actually present. The files are > downloaded again either on request or when needed. > > rsync, like all similar tools I can find, doesn't have any way of > handling these evicted files. I have been thinking about this and I > think I know how to make it work. > > The short explanation is that iCloud Drive preserves access to the > required metadata. Here is an abbreviated output from `mdls` of an > evicted file: > > ``` > % mdls .1998-tax-return.pdf.icloud > kMDItemContentCreationDate = 2018-08-28 18:32:24 +0000 > kMDItemContentCreationDate_Ranking = 2021-11-05 00:00:00 +0000 > kMDItemContentModificationDate = 2018-08-28 18:32:24 +0000 > kMDItemDateAdded = 2021-11-05 15:42:15 +0000 > kMDItemDisplayName = "1998-tax-return.pdf" > kMDItemFSContentChangeDate = 2018-08-28 18:32:24 +0000 > kMDItemFSCreationDate = 2018-08-28 18:32:24 +0000 > kMDItemFSSize = 1929932 > kMDItemInterestingDate_Ranking = 2018-08-28 00:00:00 +0000 > kMDItemLogicalSize = 1929932 > kMDItemPhysicalSize = 1929932 > ``` > > This metadata, I think, is enough to pass the rsync quick check as > described in the man page. Therefore, I suspect what needs to happen, > from a code perspective, is that rsync needs to be modified to do the > following when it finds an evicted file. All evicted files are named > consistently, `.<filename>.iCloud` so they are easy to spot. > > 1. Perform the system call equivalent of the mdls above to obtain the > appropriate dates and sizes. > 2. If the file cannot be skipped because it either has changed or a > checksum is required, perform the system call equivalent of `brctl > download <filename>` to get the file downloaded. > 3. Rsync the file > 4. If the file was downloaded by rsync, perform the system call > equivalent of `brctl evict <filename>` to remove the file to leave the > system in the same state. > > This simplistic algorithm would leave some open issues/caveats: > - It is likely that the rsync can only be run one-way from iCloud Drive > to non-iCloud Drive data storage. While it is possible it could be run > two-way research is needed on whether you?d have to download the old > file before you replace it. > - Timeouts may happen. iCloud Drive still gets stuck sometimes and there > will be non-zero time during downloads that don?t get stuck. > - There is no guarantee there is ever enough room on disk to hold a > specific file from iCloud Drive. This is likely resolved by MacOS > directly, however, this may take excessive time. > - If you thought running with checksums was slow before, strap in, > because those downloads are going to add up. > > All that said, I think this would accomplish a high percentage of what > users are looking for. > > Is this kind of a patch welcome? Is the plan sound? I haven't written C > since college (in the 90s!) and would be open to advice/mentoring. > > Thank you. > > regards, > > bex >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,