dovecot - Jul 2007 - A little OT but hopefully still related enough... Maildir Delivered mail naming problems.

Normally I wouldn't post off topic to a mailing list, but I have posted 
every where else I can think of and haven't had any success yet working 
this out.  I know there has to be a few people here with extensive 
knowledge of how mail works and maybe just a tip in the right direction 
would help me out at this point. 

I have migrated from mbox format to Maildir format in the last week. I 
used the utility mb2md.pl to convert all of our existing emails from one 
system to the next. I then modified my /etc/procmailrc file by putting 
this at the top:
MAILDIR=$HOME/Maildir
DEFAULT=$MAILDIR/

This way procmail is forced to deliver to the new format as well.

For the most part all of this is working perfectly. Procmail is 
delivering emails to their new proper folder and even extended rules in 
users own ~/.procmailrc are working well once the folder path was 
changed from $MAILDIR/whatever to $MAILDIR/.whatever/. Works great.

I am using Dovecot 1.0.1 to deliver our email through the IMAP protocol 
and with the clients we support (Thunderbird and Outlook) things are 
working very well for us.

So now to my problem. We also use Usermin for our users to access their 
email when they don't have a supported client available to them. When 
users look at their email with Usermin their emails are out of order 
regardless of weather I have Usermin's "Read Mail" module using
IMAP to
get mail or have it read the users Maildir directly. Sorts are just all 
jacked up. Sometimes if users then click the "Date" button to sort by 
date it snaps to; sometimes it don't. I currently have multiple posts on 
the Webmin/Usermin mailing list about this problem and it really seems 
to point back to the naming scheme of our emails now.

All emails which were converted have the classic maildir format. 
UnixTime.PID.Server...
1183737827.002379.mbox:2,RS
1183737827.002388.mbox:2,RS
1183737827.002392.mbox:2,RSa
1183737827.002394.mbox:2,RS

However, all the email that is delivered after the fact is different. 
They all look like this:
_oTH.XqwlGB.gator:2,RS
_p2F.qjVlGB.gator:2,RS
_qQF.2mpjGB.gator:2,Sa
_sK.8mTlGB.gator:2,RSa
_uOG.gCtjGB.gator:2,RS
_uxG.a6lkGB.gator:2,RS
_vnF.n-2lGB.gator:2,
_x9H.llckGB.gator:2,
_z3G.k8klGB.gator:2,S
_z5G.kAilGB.gator:2,RS

I have been searching the net for a few hours and reading various 
documentation here and there and I haven't been able to find anything 
that tells me why my mail is being named this way or how to fix it. I 
can say that my test box which also runs 10.2 slamd also delivers this 
exact same way....  For the most part I'm being told that the reason why 
Usermin's sort is all screwed up (order wise) in IMAP mode is that 
messages are displayed in the order that the IMAP server reports them.  
Like wise in Maildir mode the order which they are read.  I can 
understand why this naming stuff throws everything off at this point. 

What I don't understand is why its doing this.  I mean I'm pretty sure 
there is nothing here out of the ordinary as far as my config.  I have 
searched all over the net the last few days and haven't been able to 
find anything useful. 

So in wrap up sorry for being a little OT and thanks a head of time for 
any info or light anyone can share here.
-Jesse Smillie

On Tue, 2007-07-17 at 15:00 -0400, Jesse C. Smillie
wrote:
> I have migrated from mbox format to Maildir format in the last week. I 
> used the utility mb2md.pl to convert all of our existing emails from one 
> system to the next. I then modified my /etc/procmailrc file by putting 
> this at the top:
> MAILDIR=$HOME/Maildir
> DEFAULT=$MAILDIR/
Looks OK.
> So now to my problem. We also use Usermin for our users to access their 
> email when they don't have a supported client available to them. When 
I've no idea about Usermin.
> However, all the email that is delivered after the fact is different. 
> They all look like this:
> _oTH.XqwlGB.gator:2,RS
> _p2F.qjVlGB.gator:2,RS
It's not in a standard maildir format. Maybe Procmail has changed it
recently? Maybe it's configurable? Maybe you should just get rid of
Procmail? :)
> For the most part I'm being told that the reason why 
> Usermin's sort is all screwed up (order wise) in IMAP mode is that 
> messages are displayed in the order that the IMAP server reports them.  
> Like wise in Maildir mode the order which they are read.  I can 
> understand why this naming stuff throws everything off at this point. 
When Dovecot sees new messages in maildir it sorts the new messages by
their filename and assigns UIDs in that order. Existing messages aren't
ever reordered however. So if Dovecot sees these kind of new messages
one at a time, there's no problem. But if it sees two of them, then
those two might be in wrong order.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL:
<http://dovecot.org/pipermail/dovecot/attachments/20070717/266a774a/attachment-0002.bin>

Well I don't know what it was, but something you said encouraged me to 
give up on my current version of Procmail and move to the current 
version of procmail.  It turns out that its the package of procmail that 
came with Slamd 10.2 which was version 3.15.2 causing all of this mess.  
I DLed the latest slackbuild scripts (procmail version 3.22.5) and so 
forth from Slamd_Current, made the package, upgraded the package and now 
the new emails names are being saved properly.

I also ended up just hacking together a script to rename the currently 
bad named emails from their existing weird form to something more 
compatible using the naming scheme that the mb2md.pl script uses as a 
template.  This worked out really easy to fix all the email under 
~/Maildir/new....

I still have to come up with something to rename the emails which have 
already made their way from ~/Maildir/new to ~/Maildir/cur though...  I 
want to save the info which is currently in place after the ":".... 
but
I'll figure it out.

Thanks again!:> 
-Jesse Smillie



Timo Sirainen wrote:
> On Tue, 2007-07-17 at 15:00 -0400, Jesse C. Smillie wrote:
>   
>> I have migrated from mbox format to Maildir format in the last week. I 
>> used the utility mb2md.pl to convert all of our existing emails from
one
>> system to the next. I then modified my /etc/procmailrc file by putting 
>> this at the top:
>> MAILDIR=$HOME/Maildir
>> DEFAULT=$MAILDIR/
>>     
>
> Looks OK.
>
>   
>> So now to my problem. We also use Usermin for our users to access their
>> email when they don't have a supported client available to them.
When
>>     
>
> I've no idea about Usermin.
>
>   
>> However, all the email that is delivered after the fact is different. 
>> They all look like this:
>> _oTH.XqwlGB.gator:2,RS
>> _p2F.qjVlGB.gator:2,RS
>>     
>
> It's not in a standard maildir format. Maybe Procmail has changed it
> recently? Maybe it's configurable? Maybe you should just get rid of
> Procmail? :)
>
>   
>> For the most part I'm being told that the reason why 
>> Usermin's sort is all screwed up (order wise) in IMAP mode is that 
>> messages are displayed in the order that the IMAP server reports them.
>> Like wise in Maildir mode the order which they are read.  I can 
>> understand why this naming stuff throws everything off at this point. 
>>     
>
> When Dovecot sees new messages in maildir it sorts the new messages by
> their filename and assigns UIDs in that order. Existing messages aren't
> ever reordered however. So if Dovecot sees these kind of new messages
> one at a time, there's no problem. But if it sees two of them, then
> those two might be in wrong order.
>