A simple matter of…

Occasionally we’ve had this during a tomcat restart at a customer site.

After a little while we rebooted. Next time it occurred on a test platform and I could check shared memory (using ipcs and ipcrm commands), all files (temporary files etc). Then I restorted to “strace” which allowed the app to start.

Next time I was watching top and saw that despite having “-Xmx1536M” it still ate 3GB of memory and then fell over (this is a 32-bit userspace box so that’s the most it can have without horrible performance draining 4+4 kernels). Even -Xmx1300M it still fell over.

Now I was fairly sure it was a race condition or timing issue - and indeed, I now have a nicer workaround than rebooting (which is an admission of failure). The workaround is to drop all the caches so that the box is just as slow as after a reboot:

sync
echo 3 > /proc/sys/vm/drop_caches

We’ll also be upgrading the JVM from 1.5.0-10 to 1.5.0-14 to see if that helps.