dovecot - Aug 2007 - (slightly OT) syncing / migrating IMAP mailboxes

Hi!

I fear this may be slightly off-topic but it's both related to IMAP and to 
dovecot:

Is there any good and in paticular realiable program for synching / 
migrating one IMAP mailbox to another? The most important features for me 
is that a) no mails are lost / left out silently and b) the porgram is 
able to sync the complete mailbox including all folders without just 
giving up in the middle of the transfer. If it's somehow possible, I'd 
like to migrate directly from one IMAP account to the other, without 
running the synchronizer on one of the servers and without accessing one 
of the Maildirs directly.

I simply want to migrate my existing IMAP account from one Dovecot server 
(running 1.0beta5) to a new one (running 1.0rc15, as that's the version 
in Debian stable). Actually, a large part of the mailbox including most / 
all folders is already present on both servers, but I recently noticed 
that a few thousand mails are still missing.

The current mailbox state is the result of two earlier (obviously 
incomplete :-(   ) migrations a few months ago, from the original Dovecot 
1.0beta5 server to a Cyrus server, and - shortly thereafter - from there 
on to the currently used Dovecot 1.0rc15.

So far I've tried offlineimap, imapsync and mailsync, none of which seems 
to be really usuable. :-(

The current state with the incomplete mailbox was caused by mailsync. When 
I did the first two migrations, mailsync looked like the only program 
which worked, but seems to have "forgotten" the missing messages.
Another
extremely suspicious thing is that mailsync reports "phantom changes"
if
run again immediately after synching a folder and will start to delete 
and copy / "sync" dozends of messages even if none of the folders 
actually changed after synching.

I've read the name "imapsync" on this list several times, but I
just tried
it and just tried version 1.219. It happily starts synching the mailboxes 
and copying messages, but stops after partially synching the first 
folder - it simply exists, leaving no informational or error message 
whatsoever... Restarting imapsync will not help, it will appear to start 
synching the first folder again, but silently exit without doing 
anything.
Some of you seem to use imapsync as a migration tool for production 
systems, how do you cope with these situations and how do you ensure a 
mailboxs' migration actually completed successfully? As you're probably 
not playing Roulette with your customers' or companie's eMails, I'm 
interested in your receipts to successfully use imapsync myself.

As the third application I tried offlineimap (4.0.x-branch). During the 
migration a few months ago offlineimap (4.0.16 IIRC, synching from a 
Maildir to a remote IMAP account) was unusuable as it continuously and 
reproducibly threw exceptions at me during syncing obviously caused by 
specific messages in the accounts. Unfortunately, I wasn't able ti get 
any successful help on the offlineimap mailing list this time. :-(

Now, synching from one IMAP account to another, specifiying the new server 
as the "local" account, offlineimap did not throw exceptions caused by
mail messages so far, but it's performance is unusuable. It seems to use 
the "local" IMAP account on my target sevrer in a way which causes 
Dovecot to utilize all CPU power, copying one message every 10 to 20 
seconds - this way, a sync would take ages and frequently aborts because 
of connection problems. The target server is an Athlon64 3700+ machine...

So, does anyone have any pratical solution on how to migrate one IMAP 
mailbox to another? For various reasons I'd really prefer to go with a 
tested and working, reliable solution instead of starting to debug 
complex non-working programs I do not halfway understand and for which I 
cannot seem to get reliably support from its authors and/or its user 
community... :-/

Thanks a lot in advance for any pointers to a solution,

  Gunter

-- 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
"The obvious mathematical breakthrough would be development of an easy 
way to factor large prime numbers."        -- Bill Gates from The Road 
Ahead, p265
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+                   PGP-verschl?sselte Mails bevorzugt!                 +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part.
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070828/7a35a4b2/attachment-0002.bin>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 28 Aug 2007, Gunter Ohrner wrote:

I used UW-Imap's mailutil with success, but to copy an existing Mailbox to 
a fresh one using just IMAP. This is not your case.
> So, does anyone have any pratical solution on how to migrate one IMAP
> mailbox to another? For various reasons I'd really prefer to go with a
Well, the first thing is how do you want to decide, which mails are 
"identical".

I have great success with Maildir-based mailboxes by simply running a file 
dupe checker across both mailboxes and delete the duplicates from the 
"old" one. There was pretty few stuff left.

Way back, when we had some mboxes among the Maildirs, I transformed the 
mbox to Maildir, crafted a perl script that extracted the message ID and 
- From line from the mails and deleted any dup from the "old" mailbox
(and
compared the body before, just to be sure).
> cannot seem to get reliably support from its authors and/or its user
> community... :-/
Be yourself.

You have to make your own rules about what you consider as "equal" 
messages, e.g. when the two mailbox syncs lay one month apart:

+ The user moved a message to another folder in the "new" mail store.
+ The user has the message multiple times in his mail folders. (Even 
within the _same_ folder). This can happen intenionally and not.
+ The "same" message arrived the user multiple times through different
hops - the body and the essential headers (From, MessageID, To, ...) are 
the same, but not Received and "custom" headers (e.g. Virus scan
notices).

I would not rely on another tool.

Bye,

- -- 
Steffen Kaiser
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iQEVAwUBRtPOEi9SORjhbDpvAQJtrwgA072VxCWbV3lAokL++vCQoKb5WQL1dNK9
6dxjsg/Pj4CiAoAUOQAL1kwtVu92wUQNKRIxH5zWcMcjtFWdXUAs6ApZ8CZ/JfRC
KRMTQPTrGP8uOZAtHEryAbqcOBIdrotJeBml2XjDNlE/YLqJTBVQ4NM2SsDYAOx5
iPUd3W9U0eS3JaGtIILHtZkT2XIo+ZJZLvxAVWI32XxIvMuEL7q1dCbh6+JKJMJ9
BhEphNExx/CqE7GOnYhbYOZNxV6F0iA4OyP9qVTC5llW4X2OzPsypMBQ5fPKVSv4
2x8DmRDSmNUNoDRKad8HfjbsUfhaLzMmS3LNonQ9hqQguNBghEkF8Q==S8A0
-----END PGP SIGNATURE-----