Weblogic Memory Problem

[Vasu]

Hi..

We are getting the below error in wl-domain.log

####<22-Sep-2009 01:41:07 o'clock BST> <Warning> <WebLogicServer> <ukrt2-app06> <gc3-ukrt1-app06> <GC event thread> <<WLS Kernel>> <> <BEA-000335> <Percent of starting server free memory is now at "4%". This is less than the configured threshold of "5%". We will try to garbage collect, but you may consider rebooting.>

After this, system became unresponsive and expected a restart of application. Do we need to increase the heap settings? Thanks a lot for all the help.

Below mentioned the server configuration:

1. OTM V55 CU4 RU2 in weblogic 8.1, Linux 2.6.9.
2. Version of Java: BEA JRockit(R) (build R26.4.0-63-63688-1.4.2_11-20060626-2259-linux-ia32, )

# Java Additional Parameters
wrapper.java.additional.1=-jrockit
wrapper.java.additional.2=-Xms1800m
wrapper.java.additional.3=-Xmx1800m

[nick.polites]

Re: Weblogic Memory Problem

This could be just standard debug. You'll need to look at your Weblogic console.log file to see what your before/after GC collections were to see if you were running out of memory. If your before/after numbers are close to the same then your were running out of memory and some process was bottlenecking the system.

Have you taken 6 thread dumps, 30 seconds apart prior to the restart? This is critical for development to identify what may have caused the problem.

You can also go up to a max of 2400 for the JVM but that won't resolve this particular issue.

Edit: You may also wnat to upgrade the JVM to 1.4.2_12 as I have seen some issues with 11.

[Vasu]

Re: Weblogic Memory Problem

Thanks a lot Nick.

We took thread dumps and sent that to Oracle for their word. Will keep posted with the findings. Thanks.

[Vasu]

Re: Weblogic Memory Problem

Hi Nick,
Console.log for this time period. Do you see any problems ? Thanks for the help.
INFO | jvm 1 | 2009/09/22 01:41:34 | [memory ] 173633.657-173636.075: GC 1797682K->1725882K (1843200K), 2418.267 ms
INFO | jvm 1 | 2009/09/22 01:41:37 | [memory ] 173636.589-173639.692: GC 1843200K->1732596K (1843200K), 3103.427 ms
INFO | jvm 1 | 2009/09/22 01:41:40 | [memory ] 173640.373-173642.692: GC 1843200K->1731011K (1843200K), 2319.121 ms
INFO | jvm 1 | 2009/09/22 01:41:43 | [memory ] 173643.093-173645.579: GC 1843200K->1736341K (1843200K), 2485.914 ms
INFO | jvm 1 | 2009/09/22 01:41:46 | [memory ] 173646.088-173648.477: GC 1843200K->1727700K (1843200K), 2389.445 ms
INFO | jvm 1 | 2009/09/22 01:41:50 | [memory ] 173649.036-173652.572: GC 1843200K->1732644K (1843200K), 3535.986 ms
INFO | jvm 1 | 2009/09/22 01:41:53 | [memory ] 173652.981-173655.258: GC 1843200K->1730419K (1843200K), 2276.906 ms
INFO | jvm 1 | 2009/09/22 01:41:58 | [memory ] 173655.767-173660.100: GC 1843200K->1736210K (1843200K), 4332.806 ms

[nick.polites]

Re: Weblogic Memory Problem

Yep you definitely had a problem with a process. The GC means that Weblogic is performing full garbage collections (as opposed to nursery GC). Usually after a full GC you should get a few nursery GCs before another full one. On a normal GC, the before number should be close to what you have allocated for your JVM 1843200K the problem is that your after GC is 1736210K which should be much lower. So some process is chewing all of the memory.

The most common issue that cause this are agents but the thread dumps should tell you exactly what caused it.

[Vasu]

Re: Weblogic Memory Problem

Thanks a lot Nick for the findings. Will keep all posted once we get details from Oracle.