Very unstable appserver since Linux move

[Kristof Stevens]

Hi,

Our customer is facing serious performance issues since we changed our platform from HP-UX to Red Hat Linux

java version "1.4.2_11"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_11-b06)
BEA JRockit(R) (build R26.4.0-63-63688-1.4.2_11-20060626-2259-linux-ia32, )

# Java Additional Parameters
wrapper.java.additional.1=-jrockit
wrapper.java.additional.2=-Xms2000m
wrapper.java.additional.3=-Xmx2000m
wrapper.java.additional.4=-Xgcprio:throughput
wrapper.java.additional.5=-Xverbose:memory
wrapper.java.additional.6=-Xnoclassgc
wrapper.java.additional.7=-Dglog.home=%GLOG_HOME%/glog/config
wrapper.java.additional.8=-Duser.home=%GLOG_HOME%/glog/config
wrapper.java.additional.9=-Djava.io.tmpdir=%GLOG_HOME%/temp
wrapper.java.additional.10=-Djava.awt.headless=true
wrapper.java.additional.11=-Djava.security.auth.login.config=%GLOG_HOME%/glog/config/gc3_jaas.config
wrapper.java.additional.12=-Dweblogic.Name=gc3-141
wrapper.java.additional.13=-Dbea.home=/opt/otm/bea
wrapper.java.additional.14=-Dweblogic.home=%WL_HOME%/server
wrapper.java.additional.15=-Dweblogic.management.username=system
wrapper.java.additional.16=-Dweblogic.management.password=%WL_PW%
wrapper.java.additional.17=-Dweblogic.ProductionModeEnabled=true
wrapper.java.additional.18=-Dweblogic.management.discover=false
wrapper.java.additional.19=-Djava.security.policy=%WL_HOME%/server/lib/weblogic.policy
wrapper.java.additional.20=-Xss256k
wrapper.java.additional.21=-Xmanagement

Check console.log :

You will notice JVM is not able to release his memory and doing GC all the time :

INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.712: parallel nursery GC 2047465K->2047429K (2048000K), 35.372 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.748: parallel nursery GC 2047469K->2047445K (2048000K), 34.852 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.784: parallel nursery GC 2047484K->2047449K (2048000K), 35.232 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.820: parallel nursery GC 2047488K->2047457K (2048000K), 38.163 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.859: parallel nursery GC 2047496K->2047460K (2048000K), 35.399 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.895: parallel nursery GC 2047499K->2047462K (2048000K), 35.305 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.930: parallel nursery GC 2047502K->2047468K (2048000K), 35.339 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.967: parallel nursery GC 2047507K->2047490K (2048000K), 36.011 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26825.003: parallel nursery GC 2047530K->2047496K (2048000K), 35.725 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26825.040: parallel nursery GC 2047536K->2047513K (2048000K), 35.251 ms
INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26825.076: parallel nursery GC 2047552K->2047534K (2048000K), 36.380 ms
INFO

How to solve this issue

Thx,

Kristof

[nick.polites]

Re: Very unstable appserver since Linux move

Kristof,

While this is occurring I would suggest that you run 5 thread dumps each 30 seconds apart and see what is running. Some process is consuming too much memory when it is running and is usually caused by either a bad agent or a bad bulk plan.

Nick

[chrisplough]

Re: Very unstable appserver since Linux move

Kristof,

In addition to the assistance that Nick is giving, I'd recommend bumping up to 2200MB for your WebLogic heap -- as a large part of the issue appears to be Full GC related. I've had a lot of luck on Linux with heaps from 2000MB to 2200MB, though larger helps sometimes run out of native memory. Nick, however, has been successful on some very large sites with heaps as large as 2400MB. I guess it depends on whether your Red Hat Linux OS is 32 or 64-bit and what else may be running on the system.

Scaling back on your OTM threads may also help this, as the system will grind to a halt when all available heap memory is exhausted.

Thanks!
--Chris

[Kristof Stevens]

Re: Very unstable appserver since Linux move

Could following weblogic settings in the conf file cause this issue?

# Default timeout values.
# Do not change these without understanding their purpose.
wrapper.cpu.timeout=120
wrapper.ping.timeout=0
wrapper.startup.timeout=0
wrapper.shutdown.timeout=0
wrapper.restart.delay=5

We changed these settings to prevent the Wrapper is going to stop/startup the appserver automatically.

What about the following settings I'm currently testing on one of our testboxes? Any experience with ?

wrapper.java.additional.22=-XXtlasize64k
wrapper.java.additional.23=-XXlargeobjectlimit8k
and
wrapper.java.additional.4=-Xgcarallel



# Java Additional Parameters
wrapper.java.additional.1=-jrockit
wrapper.java.additional.2=-Xms2048m
wrapper.java.additional.3=-Xmx2048m
wrapper.java.additional.4=-Xgcarallel
wrapper.java.additional.5=-Xverbose:memory
wrapper.java.additional.6=-Xnoclassgc
wrapper.java.additional.7=-Dglog.home=%GLOG_HOME%/glog/config
wrapper.java.additional.8=-Duser.home=%GLOG_HOME%/glog/config
wrapper.java.additional.9=-Djava.io.tmpdir=%GLOG_HOME%/temp
wrapper.java.additional.10=-Djava.awt.headless=true
wrapper.java.additional.11=-Djava.security.auth.login.config=%GLOG_HOME%/glog/config/gc3_jaas.config
wrapper.java.additional.12=-Dweblogic.Name=gc3-fikylx25
wrapper.java.additional.13=-Dbea.home=/opt/otm/bea
wrapper.java.additional.14=-Dweblogic.home=%WL_HOME%/server
wrapper.java.additional.15=-Dweblogic.management.username=system
wrapper.java.additional.16=-Dweblogic.management.password=%WL_PW%
wrapper.java.additional.17=-Dweblogic.ProductionModeEnabled=true
wrapper.java.additional.18=-Dweblogic.management.discover=false
wrapper.java.additional.19=-Djava.security.policy=%WL_HOME%/server/lib/weblogic.policy
wrapper.java.additional.20=-Xss256k
wrapper.java.additional.21=-Xmanagement
wrapper.java.additional.22=-XXtlasize64k
wrapper.java.additional.23=-XXlargeobjectlimit8k
Regards,

Kristof

[chrisplough]

Re: Very unstable appserver since Linux move

Kristof,

Looking at your settings, I have the following comments:

wrapper.cpu.timeout=120
CP> I understand why you changed this one -- though I hope you don't have many GCs that take longer than 60-90 seconds. JRockit is very fast, even at full GCs, so if this is the case, than either your server is swapping, your heap is too small (I'd move up to 2200MB), or you're running too much in the system at once and need to throttle it, in order to increase performance. This may not seem intuitive, but if you throttle and reduce JVM heap memory usage, JRockit wlll be able to process
wrapper.startup.timeout=0
CP> This should be 60 - no reason to change.
wrapper.shutdown.timeout=0
CP> This should be 120. If WebLogic doesn't shutdown in 120 seconds, there is an issue and JSW should kill it. This setting ensures that WebLogic is always shutdown before the next one starts up and you don't end up with hung or zombie processes.
wrapper.java.additional.22=-XXtlasize64k
CP> I honestly haven't had to use this one in the past. I've seen it referenced in a developer's blog in the past, but haven't had a system where it was necessary to utilize. (Henrik Stahl's Blog: June 2006 Archive)wrapper.java.additional.23=-XXlargeobjectlimit8k
CP> I honestly haven't had to use this one in the past. I've seen it referenced in a developer's blog in the past, but haven't had a system where it was necessary to utilize. (Henrik Stahl's Blog: June 2006 Archive)wrapper.java.additional.4=-Xgcarallel
CP> I've had much better luck with "-Xgcprio:throughput", as this allows JRockit to dynamically switch GC algorithms while running -- to ensure that the JVM adapts to the current usage and load.

Honestly, I believe that your greatest gains would be from increasing the heap, and adjusting the workload so that the JVM heap doesn't get overwhelmed -- thus avoiding full GCs and GC memory lock issues; and thereby creating a much faster system.

Hope this helps!

--Chris