rsync - Jul 2003 - compression and encryption

Hello folks!

I've been thinking about adding the capability to store files compressed 
and/or encrypted on either side of the rsync transfer. Let's say there's
a storage space provider. I want to store my files on that server 
compressed (so I don't use more paid space than I need), and encrypted 
(so they can't read my data).

I don't know much about encryption, but I suppose that there are some 
ciphers that are reasonably strong and don't have the same problem like 
gzip (that a single changed byte in the middle of the file affects the 
contents of the rest of the file)? Even if not, the same thing like with 
gzip (--rsyncable) could probably be done. The goal is to do the 
encryption on one side so that the storage provider doesn't ever see the 
unencrypted content or the key.

As for compression, I was thinking of making rsync temporarily 
uncompress the file, then update it, and then recompress it. This way, 
the gzip rsyncable patch wouldn't be needed, and one could actually use 
any other compressor, like bzip2.

One very general way to achieve things like this would be to add the 
ability to filter every file through a given filter before and after the 
transfer, like this:

rsync --filter-source="encrypt --key=xyzzy"
--filter-dest-before=bunzip2
--filter-dest-after=bzip2 /source/directory host:

It would be extremely slow, of course, but it may be worth it in some 
situations. (The --filter-source wouldn't actually need to be so slow if 
the filter was guaranteed to produce exactly as many bytes as it consumes.)

Of course, there's at least one major problem with this: generally, the 
server can't let the client run any command as the filter. But the 
server side rsync could be modified to have a config file option with a 
regexp for allowed commands, or something like that.

Obviously, rsync could also be amended with built-in encryption. Less 
general solution, but worth considering, too.

Any thoughts or opinions? Any existing work that I should be aware of? 
Would a patch in some of the directions outlined above be accepted?

Thanks,

Vaclav Dvorak

On Mon, Jul 28, 2003 at 05:33:13AM +0200, Vaclav Dvorak
wrote:
> Hello folks!
> 
> I've been thinking about adding the capability to store files
compressed
> and/or encrypted on either side of the rsync transfer. Let's say
there's
> a storage space provider. I want to store my files on that server 
> compressed (so I don't use more paid space than I need), and encrypted 
> (so they can't read my data).
[snip]
> 
> Any thoughts or opinions? Any existing work that I should be aware of? 
> Would a patch in some of the directions outlined above be accepted?
Read the list archives.  This has been talked to death.

Most recently, Martin Langhoff has spoken of doing some work
on using sender-side filtering that sounds promising.  There
is also a receiver-side patch out there with the performance
issues you mention.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

> I've been thinking about adding the capability to store files
compressed
> and/or encrypted on either side of the rsync transfer. Let's say
there's
> a storage space provider. I want to store my files on that server 
> compressed (so I don't use more paid space than I need), and encrypted 
> (so they can't read my data).
>
> [snip]
>
> Any thoughts or opinions? Any existing work that I should be aware of? 
You could look at BackupPC, http://backuppc.sourceforge.net, which does
compression of server side files with rsync.  It stores file attributes
separate from the compressed files, so it can compare the correct size
to the sender's side file list.  It also only stores identical files
once, including files with different names or from different machines.

Craig