dovecot - Dec 2012 - Automatically Cleaning Kerberos Credential Cache Files

I'm in a situation here at work where I'm trying to support a mixed 
network of OS X and RHEL desktop machines with a Postfix/Dovecot 
combination.
   - user account information is stored in LDAP
   - user credentials are in MIT Kerberos
   - server is running RHEL 6/Dovecot 2.0.9/Postfix 2.6.6

I am currently using the PAM passdb module to authenticate my users (I 
began to have trouble with using GSSAPI directly). After I implemented 
it, a few weeks later, I noticed that some users were no longer getting 
their mail if they hadn't logged in during the past day. Postfix's mailq
showed that hundreds of messages were backing up in the queue. I 
eventually tracked it down to leftover Kerberos credential cache files 
(/tmp/krb5cc_????) sitting in /tmp on the mail server. The presence of 
expired credential files was preventing Postfix from delivering mail to 
those users' mail spools. If I delete the credential files manually, 
Postfix immediately delivers the queued emails.

Currently, I have a cron job deleting the files manually every night. 
Obviously, this is a cruddy solution. I have Dovecot configured on a 
RHEL 6 box. The Pam stack on a RHEL 6 machine uses sssd (pam_sss.so) for 
authentication with Kerberos, not pam_krb5.so. I'm trying to track down 
which piece of the puzzle is responsible for cleaning up leftover 
credential caches. Is there a configuration option I can pass to 
Dovecot's passdb directly to clean up these cache files? Do others 
generally have more success using a custom PAM stack with pam_krb5.so 
instead of pam_sss.so? Poring over sssd's configuration options didn't 
reveal anything useful. I'm still not sure why Postfix even cares if 
there are expired credential cache files in /tmp at all.

I'm back to trying GSSAPI directly again as well in the meantime on a 
few test clients. When I used that in the past, users were getting 
issued duplicate Kerberos tickets and users were forced to log into the 
mail server directly using SSH after a day in order to get their mail to 
work (seemingly a related issue). If I get those same errors again, I'll 
start another thread.

The output of doveconf -N:

# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.14.1.el6.x86_64 x86_64 Red Hat Enterprise Linux 
Server release 6.3 (Santiago) ext4
auth_cache_size = 10 k
auth_cache_ttl = 5 hours
auth_default_realm = server.com
auth_gssapi_hostname = hostname.server.com
auth_krb5_keytab = /etc/dovecot/dovecot.keytab
auth_mechanisms = gssapi plain
auth_realms = server.com
auth_username_format = %n
auth_verbose = no
disable_plaintext_auth = yes
listen = *
mail_location = maildir:/var/spool/mail/%u
mail_privileged_group = mail
mbox_read_locks = fcntl
mbox_write_locks = fcntl
namespace {
   inbox = yes
   location    prefix    separator = /
   type = private
}
passdb {
   driver = pam
}
protocols = pop3 imap
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0666
     user = postfix
   }
}
ssl = yes
ssl_cert = </usr/share/ssl/private/mail.server.com.pem
ssl_key = </usr/share/ssl/private/mail.server.com.pem
userdb {
   driver = passwd
}
protocol imap {
   imap_client_workarounds = tb-extra-mailbox-sep
   mail_plugins = " listescape"
}

Thanks for any pointers,
Maura Dailey
maura at eclipse.ncsc.mil

At  3PM -0500 on 10/12/12 you (Maura Dailey) wrote:
> I'm in a situation here at work where I'm trying to support a mixed
> network of OS X and RHEL desktop machines with a Postfix/Dovecot 
> combination.
>    - user account information is stored in LDAP
>    - user credentials are in MIT Kerberos
>    - server is running RHEL 6/Dovecot 2.0.9/Postfix 2.6.6
> 
> I am currently using the PAM passdb module to authenticate my users (I 
> began to have trouble with using GSSAPI directly). After I implemented 
> it, a few weeks later, I noticed that some users were no longer getting 
> their mail if they hadn't logged in during the past day. Postfix's
mailq
> showed that hundreds of messages were backing up in the queue. I 
> eventually tracked it down to leftover Kerberos credential cache files 
> (/tmp/krb5cc_????) sitting in /tmp on the mail server. The presence of 
> expired credential files was preventing Postfix from delivering mail to 
> those users' mail spools. If I delete the credential files manually, 
> Postfix immediately delivers the queued emails.
This is rather odd. Is krb5-authenticated NFS involved here, or does
Postfix's delivery make any other use of Kerberos? The only other thing
I can think of is that so many expired ccaches are accumulating that the
user goes over their inode quota.
> Currently, I have a cron job deleting the files manually every night. 
> Obviously, this is a cruddy solution.
Well, I don't know about that: obviously, it would be a good idea to
work out what's going on here, in case it causes anything else to go
wrong, but a cronjob clearing out /tmp is a good idea in any case.
> I have Dovecot configured on a 
> RHEL 6 box. The Pam stack on a RHEL 6 machine uses sssd (pam_sss.so) for 
> authentication with Kerberos, not pam_krb5.so. I'm trying to track down
> which piece of the puzzle is responsible for cleaning up leftover 
> credential caches. Is there a configuration option I can pass to 
> Dovecot's passdb directly to clean up these cache files?
There are two relevant Dovecot settings: passing setcred=yes to the pam
passdb will make Dovecot call pam_setcred, and passing session=yes will
make it open and immediately close a PAM session. (Probably it ought to
wait until the IMAP user logs out to close the session, but currently it
can't do that because of the way the passdb lookups are done.) Changing
either of these may have an effect, depending on when exactly your PAM
module creates and destroys ccaches.
> Do others 
> generally have more success using a custom PAM stack with pam_krb5.so 
> instead of pam_sss.so?
Well, I don't use RH (I use FreeBSD), but I use and would recommend Russ
Allbery's pam_krb5.so, which may or may not be the same as the normal
pam_krb5.so provided by your system. It has options to control whether
and where ccaches are created; assuming Dovecot doesn't need krb5 creds
(say, for NFS), you would probably be better off telling it not to
create a permanent ccache at all.

http://www.eyrie.org/~eagle/software/pam-krb5/
> Poring over sssd's configuration options didn't 
> reveal anything useful. I'm still not sure why Postfix even cares if 
> there are expired credential cache files in /tmp at all.
> 
> I'm back to trying GSSAPI directly again as well in the meantime on a 
> few test clients. When I used that in the past, users were getting 
> issued duplicate Kerberos tickets and users were forced to log into the 
> mail server directly using SSH after a day in order to get their mail to 
> work (seemingly a related issue). If I get those same errors again,
I'll
> start another thread.
This certainly does all sound related. What are the ccache files called:
are they just /tmp/krb5cc_UID or is there a random portion as well? Are
they being created with the correct permissions, and are there any
security policies (SELinux or ACLs of some kind) set up which might
interfere with their creation or destruction? 

What happens if you log in as an ordinary user (preferably using the
same PAM stack as Dovecot uses), use klist to find the ccache name and
'ls -i' to find its inode number, then manually kinit again? Does the
kinit succeed, and does the new ccache have a different inode number
from before? Does the ccache file get removed when you log out?

Ben

On 12/10/2012 05:31 PM, Ben Morrow wrote:
> At  3PM -0500 on 10/12/12 you (Maura Dailey) wrote:
>> I'm in a situation here at work where I'm trying to support a
mixed
>> network of OS X and RHEL desktop machines with a Postfix/Dovecot
>> combination.
>>     - user account information is stored in LDAP
>>     - user credentials are in MIT Kerberos
>>     - server is running RHEL 6/Dovecot 2.0.9/Postfix 2.6.6
>>
>> I am currently using the PAM passdb module to authenticate my users (I
>> began to have trouble with using GSSAPI directly). After I implemented
>> it, a few weeks later, I noticed that some users were no longer getting
>> their mail if they hadn't logged in during the past day.
Postfix's mailq
>> showed that hundreds of messages were backing up in the queue. I
>> eventually tracked it down to leftover Kerberos credential cache files
>> (/tmp/krb5cc_????) sitting in /tmp on the mail server. The presence of
>> expired credential files was preventing Postfix from delivering mail to
>> those users' mail spools. If I delete the credential files
manually,
>> Postfix immediately delivers the queued emails.
> This is rather odd. Is krb5-authenticated NFS involved here, or does
> Postfix's delivery make any other use of Kerberos? The only other thing
> I can think of is that so many expired ccaches are accumulating that the
> user goes over their inode quota.
Each user has one credential cache file in /tmp on the mail server after 
logging into Dovecot. We aren't using randomized names, so everything is 
in the standard format /tmp/krb5cc_uid. We do use KRB5 authenticated 
(and encrypted) NFS, but we don't deliver mail to home directories. 
Since all users are "real" users, and our office size is small,
everyone
has a mail spool directory on the mail server.

Postfix is configured to use Dovecot for Kerberos. The relevant lines in 
its /etc/postfix/main.cf are as follows:
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_recipient_restrictions = permit_mynetworks, 
permit_sasl_authenticated, reject_unauth_destination
smtpd_client_restrictions = permit_mynetworks, 
permit_sasl_authenticated, reject
smtpd_sasl_security_options = noanonymous
>> Currently, I have a cron job deleting the files manually every night.
>> Obviously, this is a cruddy solution.
> Well, I don't know about that: obviously, it would be a good idea to
> work out what's going on here, in case it causes anything else to go
> wrong, but a cronjob clearing out /tmp is a good idea in any case. 
Fair enough. And of course, I get a cache file every time I use sudo or 
log in through a GUI. Those aren't cleaned up either! Maybe I should be 
deleting credential cache files on all the machines every night. Those 
cache files have never given me a problem, though.
>> I have Dovecot configured on a
>> RHEL 6 box. The Pam stack on a RHEL 6 machine uses sssd (pam_sss.so)
for
>> authentication with Kerberos, not pam_krb5.so. I'm trying to track
down
>> which piece of the puzzle is responsible for cleaning up leftover
>> credential caches. Is there a configuration option I can pass to
>> Dovecot's passdb directly to clean up these cache files?
> There are two relevant Dovecot settings: passing setcred=yes to the pam
> passdb will make Dovecot call pam_setcred, and passing session=yes will
> make it open and immediately close a PAM session. (Probably it ought to
> wait until the IMAP user logs out to close the session, but currently it
> can't do that because of the way the passdb lookups are done.) Changing
> either of these may have an effect, depending on when exactly your PAM
> module creates and destroys ccaches. 
I've tried session=yes by itself, but I don't think I've tried it
with
setcred=yes. I'll throw it in there and give it a try after I send this 
email.
>> Do others
>> generally have more success using a custom PAM stack with pam_krb5.so
>> instead of pam_sss.so?
> Well, I don't use RH (I use FreeBSD), but I use and would recommend
Russ
> Allbery's pam_krb5.so, which may or may not be the same as the normal
> pam_krb5.so provided by your system. It has options to control whether
> and where ccaches are created; assuming Dovecot doesn't need krb5 creds
> (say, for NFS), you would probably be better off telling it not to
> create a permanent ccache at all.
>
> http://www.eyrie.org/~eagle/software/pam-krb5/
I haven't had to configure pam_krb5.so directly before (we use the Red 
Hat/Fedora configured default, pam_sss.so, which claims to be a one stop 
shop for LDAP/Kerberos/NIS,etc.), but it does seem to have more options. 
We certainly don't need credential caches to stick around for email 
users. All the mail spools are stored on locally mounted storage on the 
mail server.
>> Poring over sssd's configuration options didn't
>> reveal anything useful. I'm still not sure why Postfix even cares
if
>> there are expired credential cache files in /tmp at all.
>>
>> I'm back to trying GSSAPI directly again as well in the meantime on
a
>> few test clients. When I used that in the past, users were getting
>> issued duplicate Kerberos tickets and users were forced to log into the
>> mail server directly using SSH after a day in order to get their mail
to
>> work (seemingly a related issue). If I get those same errors again,
I'll
>> start another thread.
> This certainly does all sound related. What are the ccache files called:
> are they just /tmp/krb5cc_UID or is there a random portion as well? Are
> they being created with the correct permissions, and are there any
> security policies (SELinux or ACLs of some kind) set up which might
> interfere with their creation or destruction?
No random portion (the only application we have configured with random 
ccache names is SSH), just the usual /tmp/krb5cc_uid. Permissions appear 
correct, they belong to the appropriate user and group accounts. The 
SELinux permissions are set to: system_u:object_r:user_tmp_t:s0. This is 
different from the machine's credential cache 
(system_u:object_r:gssd_tmp_t:s0), but I've disabled and re-enabled 
SELinux during different parts of my testing and didn't notice any errors.

Using GSSAPI directly doesn't create any cached credential files on the 
server. However, I just verified that I'm still sporadically getting the 
duplicate tickets (two lines for imap/hostname.server.com at SERVER.COM, 
with identical expiration dates). It comes and goes. I have a few users 
testing it for me, and they'll let me know if their logins break 
tomorrow like they have in the past once the normal log in period 
elapses. Since the "fix" for that problem is to log into the mail 
server, I have a hard time testing for the problem myself (being logged 
into the mail server nearly continuously just now).
> What happens if you log in as an ordinary user (preferably using the
> same PAM stack as Dovecot uses), use klist to find the ccache name and
> 'ls -i' to find its inode number, then manually kinit again? Does
the
> kinit succeed, and does the new ccache have a different inode number
> from before? Does the ccache file get removed when you log out?
I've noticed that the ccache files for regular users do not get deleted 
when they log out. On another user's workstation, I found a ccache file 
for my account from a week ago (my current session was using a randomly 
generated SSH ccache file). I ran a command with sudo to force it to 
reauthenticate with pam_sss, and the timestamp on the ccache file was 
updated and the inode changed. My best guess is that pam_sss.so just 
doesn't do ccache cleanup.
> Ben
>
Thanks for taking a look, I appreciate all the suggestions. This 
problem's been driving me up a wall for weeks now. I only just managed 
to narrow down the cause in the last few days. If I can't get a fix 
working in Dovecot, I'll try tackling it from a different perspective. I 
understand why Dovecot might not be cleaning up credential files 
(especially if it's just calling PAM normally), but I definitely do not 
understand why Postfix is behaving the way it does. If I can just get 
GSSAPI to work, I could probably bypass the problem as well.

- Maura Dailey
maura at eclipse.ncsc.mil

Ben Morrow writes:
> In any case, it's likely that the delivery process looks in the
user's
> home directory even if delivery is to a separate mail spool, unless
> you've taken steps to prevent this.
I can confirm this.  I think the reason is that dovecot changes to the
user's home directory so that it has a place to dump core.  This causes
problem for my setup since I user per-user NFS mounts, so a delivery to an
internal mailing list caused hundreds of NFS automounts.

I patched this behaviour out, but there is probably a more elegant solution
(unset MAIL_STORAGE_SERVICE_FLAG_ENABLE_CORE_DUMPS? How?).

Joseph Tam <jtam.home at gmail.com>

--- ./src/lib-storage/mail-storage-service.c~	Sat May 19 08:42:12 2012
+++ ./src/lib-storage/mail-storage-service.c	Mon Jun 25 22:00:57 2012
@@ -615,21 +615,21 @@

-	if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP) != 0
&&
-	    (user->flags & MAIL_STORAGE_SERVICE_FLAG_ENABLE_CORE_DUMPS) == 0) {
-		/* we don't want to write core files to any users' home
-		   directories since they could contain information about other
-		   users' mails as well. so do no chdiring to home. */
-	} else if (*home != '\0' &&
-		   (user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_CHDIR) == 0) {
-		/* If possible chdir to home directory, so that core file
-		   could be written in case we crash. */
-		if (chdir(home) < 0) {
-			if (errno == EACCES) {
-				i_error("%s", eacces_error_get("chdir",
-						t_strconcat(home, "/", NULL)));
-			} if (errno != ENOENT)
-				i_error("chdir(%s) failed: %m", home);
-			else if (mail_set->mail_debug)
-				i_debug("Home dir not found: %s", home);
-		}
-	}
+//	if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_TEMP_PRIV_DROP) != 0
&&
+//	    (user->flags & MAIL_STORAGE_SERVICE_FLAG_ENABLE_CORE_DUMPS) == 0)
{
+//		/* we don't want to write core files to any users' home
+//		   directories since they could contain information about other
+//		   users' mails as well. so do no chdiring to home. */
+//	} else if (*home != '\0' &&
+//		   (user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_CHDIR) == 0) {
+//		/* If possible chdir to home directory, so that core file
+//		   could be written in case we crash. */
+//		if (chdir(home) < 0) {
+//			if (errno == EACCES) {
+//				i_error("%s", eacces_error_get("chdir",
+//						t_strconcat(home, "/", NULL)));
+//			} if (errno != ENOENT)
+//				i_error("chdir(%s) failed: %m", home);
+//			else if (mail_set->mail_debug)
+//				i_debug("Home dir not found: %s", home);
+//		}
+//	}