dovecot - Feb 2014 - Dovecot 2.2.10 crash / infinite loop bug

Hello,

I though this bug should be reported, although there is workaround 
already made and in use.

Dovecot 2.2.10 crashes into out of memory error when there is defined 
system users (userdb+passdb) and own virtual users (userdb+passdb). 
Removing either one solves the issue and Dovecot is working as it 
should. When memory leak (I replaced pw_init and gr_init with the 
version that is found from 2.1.17) is fixed from lib/ipwd.c source, the 
auth worker gets to infinite loop and authentication timeouts(timeout 
was increased, but worker does not end what ever it's doing). This far I 
got chance to debug this before other passdb was removed due lack of time.

Here's logs, configs and backtrace:

Feb  1 07:39:19 openbsd dovecot: auth-worker(30906): Fatal: 
pool_system_realloc(268435456): Out of memory
Feb  1 07:39:19 openbsd dovecot: auth: Error: auth worker: Aborted 
request: Worker process died unexpectedly
Feb  1 07:39:19 openbsd dovecot: auth-worker(30906): Fatal: master: 
service(auth-worker): child 30906 returned error 83 (Out of memory 
(service auth-worker { vsz_limit=256 MB }, you may need to increase it) 
- set CORE_OUTOFMEM=1 environment to get core dump)
Feb  1 07:39:19 openbsd dovecot: auth: Error: static(email at localhost): 
passdb doesn't support lookups, can't verify user's existence
Feb  1 07:39:19 openbsd dovecot: lda: Error: user email at localhost: Auth 
USER lookup failed
Feb  1 07:39:19 openbsd dovecot: lda: Fatal: Internal error occurred. 
Refer to server log for more information.
Feb  1 07:39:19 openbsd postfix/pipe[17487]: 5DBAE5F289: 
to=<email at localhost>, relay=dovecot, delay=37607, 
delays=37606/0.02/0/0.67, dsn=4.3.0, status=deferred (temporary failure)

# 2.2.10: /etc/dovecot/dovecot.conf
# OS: OpenBSD 5.5 i386  ffs
auth_debug = yes
auth_verbose = yes
first_valid_uid = 1000
imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
mail_debug = yes
mail_location = maildir:/var/vmail/mydomain/%n
mbox_write_locks = fcntl
mmap_disable = yes
namespace inbox {
   inbox = yes
   location    mailbox Spam {
     auto = subscribe
   }
   prefix }
passdb {
   driver = bsdauth
}
passdb {
   args = scheme=CRYPT username_format=%u /etc/dovecot/users
   driver = passwd-file
}
pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
protocols = imap lmtp
service auth {
   unix_listener /var/spool/postfix/private/auth {
     mode = 0666
   }
}
ssl_cert = </cert.pem
ssl_key = </cert.pem
userdb {
   driver = passwd
   override_fields = uid=vmail gid=vmail quota_rule=*:storage=1G
}
userdb {
   args = username_format=%u /etc/dovecot/users
   driver = passwd-file
   override_fields = uid=vmail gid=vmail home=/var/vmail/%d/%u 
quota_rule=*:storage=1G
}
verbose_ssl = yes
protocol imap {
   mail_plugins }

(gdb) bt full
#0  0x0221be9d in kill () at <stdin>:2
No locals.
#1  0x02285e36 in raise (s=6) at /usr/src/lib/libc/gen/raise.c:39
No locals.
#2  0x02285d5c in abort () at /usr/src/lib/libc/stdlib/abort.c:70
         p = (struct atexit *) 0x7fc2c000
         mask = 4294967263
         cleanup_called = 1
#3  0x0245dcd4 in default_fatal_finish (type=Variable "type" is not 
available.
) at failures.c:193
         backtrace = Variable "backtrace" is not available.
(gdb) bt
#0  0x0221be9d in kill () at <stdin>:2
#1  0x02285e36 in raise (s=6) at /usr/src/lib/libc/gen/raise.c:39
#2  0x02285d5c in abort () at /usr/src/lib/libc/stdlib/abort.c:70
#3  0x0245dcd4 in default_fatal_finish (type=Variable "type" is not 
available.
) at failures.c:193
#4  0x0245dd4d in i_internal_fatal_handler (ctx=0xcfbec0f4, 
format=0x22406990 "pool_system_realloc(%lu): Out of memory", 
args=0xcfbec118 "") at failures.c:657
#5  0x0245cee3 in i_fatal_status (status=Could not find the frame base 
for "i_fatal_status".
) at failures.c:295
#6  0x02475543 in pool_system_realloc (pool=Could not find the frame 
base for "pool_system_realloc".
) at mempool-system.c:120
#7  0x02464c47 in i_realloc (mem=0x86122000, old_size=134217728, 
new_size=268435456) at imem.c:14
#8  0x0246505c in pw_init () at ipwd.c:23
#9  0x024651f7 in i_getpwnam (name=0x76941140"email at localhost", 
pwd_r=0xcfbec1cc) at ipwd.c:51
#10 0x164eef8e in bsdauth_verify_plain (request=0x76941020, 
password=0x78a183a2 "testi", callback=0x164e0320 
<verify_plain_callback>) at passdb-bsdauth.c:26
#11 0x164e0bf4 in auth_worker_input (client=0x7b386460) at 
auth-worker-client.c:205
#12 0x0246f9c2 in io_loop_call_io (io=0x7d1ae8c0) at ioloop.c:388
#13 0x02470e97 in io_loop_handler_run (ioloop=0x7d1ae3c0) at 
ioloop-kqueue.c:151
#14 0x0246f928 in io_loop_run (ioloop=0x7d1ae3c0) at ioloop.c:412
#15 0x0241609d in master_service_run (service=0x7e433d00, 
callback=0x164e66a0 <worker_connected>) at master-service.c:566
#16 0x164e6cc8 in main (argc=Cannot access memory at address 0x0
) at main.c:393
Current language:  auto; currently asm

Sincerelly,
Jani Hast

On 8.2.2014, at 4.53, Jani Hast <ghast123 at gmail.com> wrote:
> Hello,
> 
> I though this bug should be reported, although there is workaround already
made and in use.
> 
> Dovecot 2.2.10 crashes into out of memory error when there is defined
system users (userdb+passdb) and own virtual users (userdb+passdb). Removing
either one solves the issue and Dovecot is working as it should.
I can't think of why that would make a difference.
> When memory leak (I replaced pw_init and gr_init with the version that is
found from 2.1.17) is fixed from lib/ipwd.c source,
I don't see any bugs or memory leaks in ipwd.c. Sounds like it could be an
OpenBSD bug.
> the auth worker gets to infinite loop and authentication timeouts(timeout
was increased, but worker does not end what ever it's doing).
Would be useful to know where it's looping (gdb -p <pid>, bt, n, n, n,
n, n, ...).
> passdb {
>  driver = bsdauth
> }
I don't have any BSDs installed, so can't easily test this.