[SOLVED] JVM Timeouts in 5.0 under Windows

[acuartero]

We're experiencing a lot of of JVM Timeout error messages (like "JVM Process has not received any CPU time for 11 seconds Extending timeouts.") in our console.log file. During this time OTM becomes unresponsive but usually does recover. These appear to happen at random but more often than not on days with heavy volume. We also have lots of memory related errors (OutOfMemory, swapDisk, and Page faults during GC) and we do know that we are running up against the limits of memory management under windows. But I don't see a direct correlation between the memory related errors and the JVM timeout errors.

My question is can we enable an additional level of logging that will help us determine what is happening inside the JVM at the time that a timeout error occurs?

I started running the EQ/TX thread diagnostics to see if I can match up the timeouts with a specific thread or event but so far haven't been lucky enough to have a timeout occur while thread diags are running.

Thanks,
Alan

[chrisplough]

Re: JVM Timeouts in 5.0 under Windows

Hey Alan!

There's no additional logging that I can recommend, besides ensuring that verbosegc is enabled. While you're not seeing a correlation between memory issues and the timeouts, I'd look a bit deeper into it. One of the issues with Win32 is that the OTM heaps are limited to about 1.5GB. There was a large UK client that we hosted back in the G-Log days and this was one of the main issues they encountered.

When the JVM either runs out of memory (large amounts of garbage collection using up all CPU time) or threads, these timeout issues are encountered. Usually the former is the cause. Oh - I forgot. I've also seen this issue if too much memory is allocated, I've seen the same thing. A 1.5GB JRockit heap can consume 2.5-3GB of total memory, due to JRockit's memory usage for optimization.

Hope this helps!
--Chris

[acuartero]

Re: JVM Timeouts in 5.0 under Windows

Thanks Chris, we have been running with the default heap size allocation (1025) and with the /3GB switch in our boot.ini at Oracle's recommendation in an attempt to give the user space as much memory as possible and reduce the amount of page faults that are occurring. Unfortunately we still appear to be running up against the limits of Windows memory management. From what you said it appears that the JVM timeouts are most probably the result of our memory management issues as well. The plan is for us to move off the Windows platform on to Linux in an attempt to solve these issues. I see you have a performance tuning guide posted that includes some Linux tuning hints, these should come in handy as well.

Thanks,
Alan

[chrisplough]

Re: JVM Timeouts in 5.0 under Windows

Alan,

I agree with you 100%. While the computational performance on Linux isn't much better than Win32, the access to larger memory heaps and ease of management make it a better overall platform.

--Chris