Debugging OTM Native Heap Crashes on AIX

[chrisplough]

Hello all,

I'm currently debugging an issue for a client, who is running on AIX. I hate to say it, but this has unfortunately become one of the more troublesome platforms for OTM, both in terms of performance (Java performance is poor -- even though the processors are fantastic) and stability (Java crashes A LOT under load).

The issue we're encountering now is that the WebLogic server is randomly crashing as the JVM runs out of native memory (note -- NOT the Java Heap).

For more information on how Java allocates memory on AIX, see the following article:

• Getting more memory in AIX for your Java applications

I'll most more details as I get them, but right now, we're running svmon at regular intervals in order to understand the memory usage and determine what we can do about it.

svmon -P <weblogic_java_pid> -m -i <interval>

To determine the <weblogic_java_pid>, just run:

ps -ef | grep java | grep weblogic\.Server

The worst-case scenario is that this is a bug in how OTM and the IBM JDK work together, which causes a memory leak in the native memory area -- since it would require a code fix, which would be especially hard to track down and resolve.

Thanks,
Chris

[chrisplough]

Re: Debugging OTM Native Heap Crashes on AIX

As a quick update - here's a website that describes the things you'll want to accomplish and log before calling IBM -- otherwise they'll just make you do it anyways.

IBM - Mustgather: Gathering data for OutOfMemoryErrors on AIX

The biggest help is the script to automate the collection of svmon and other data (attached below).

AIX_memory_leak.sh.zip

Thanks,
Chris