Announcement

Collapse
No announcement yet.

Very unstable appserver since Linux move

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Very unstable appserver since Linux move

    Hi,

    Our customer is facing serious performance issues since we changed our platform from HP-UX to Red Hat Linux

    java version "1.4.2_11"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_11-b06)
    BEA JRockit(R) (build R26.4.0-63-63688-1.4.2_11-20060626-2259-linux-ia32, )

    # Java Additional Parameters
    wrapper.java.additional.1=-jrockit
    wrapper.java.additional.2=-Xms2000m
    wrapper.java.additional.3=-Xmx2000m
    wrapper.java.additional.4=-Xgcprio:throughput
    wrapper.java.additional.5=-Xverbose:memory
    wrapper.java.additional.6=-Xnoclassgc
    wrapper.java.additional.7=-Dglog.home=%GLOG_HOME%/glog/config
    wrapper.java.additional.8=-Duser.home=%GLOG_HOME%/glog/config
    wrapper.java.additional.9=-Djava.io.tmpdir=%GLOG_HOME%/temp
    wrapper.java.additional.10=-Djava.awt.headless=true
    wrapper.java.additional.11=-Djava.security.auth.login.config=%GLOG_HOME%/glog/config/gc3_jaas.config
    wrapper.java.additional.12=-Dweblogic.Name=gc3-141
    wrapper.java.additional.13=-Dbea.home=/opt/otm/bea
    wrapper.java.additional.14=-Dweblogic.home=%WL_HOME%/server
    wrapper.java.additional.15=-Dweblogic.management.username=system
    wrapper.java.additional.16=-Dweblogic.management.password=%WL_PW%
    wrapper.java.additional.17=-Dweblogic.ProductionModeEnabled=true
    wrapper.java.additional.18=-Dweblogic.management.discover=false
    wrapper.java.additional.19=-Djava.security.policy=%WL_HOME%/server/lib/weblogic.policy
    wrapper.java.additional.20=-Xss256k
    wrapper.java.additional.21=-Xmanagement

    Check console.log :

    You will notice JVM is not able to release his memory and doing GC all the time :

    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.712: parallel nursery GC 2047465K->2047429K (2048000K), 35.372 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.748: parallel nursery GC 2047469K->2047445K (2048000K), 34.852 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.784: parallel nursery GC 2047484K->2047449K (2048000K), 35.232 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.820: parallel nursery GC 2047488K->2047457K (2048000K), 38.163 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.859: parallel nursery GC 2047496K->2047460K (2048000K), 35.399 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.895: parallel nursery GC 2047499K->2047462K (2048000K), 35.305 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.930: parallel nursery GC 2047502K->2047468K (2048000K), 35.339 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26824.967: parallel nursery GC 2047507K->2047490K (2048000K), 36.011 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26825.003: parallel nursery GC 2047530K->2047496K (2048000K), 35.725 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26825.040: parallel nursery GC 2047536K->2047513K (2048000K), 35.251 ms
    INFO | jvm 1 | 2008/01/29 12:32:39 | [memory ] 26825.076: parallel nursery GC 2047552K->2047534K (2048000K), 36.380 ms
    INFO

    How to solve this issue

    Thx,

    Kristof

  • #2
    Re: Very unstable appserver since Linux move

    Kristof,

    While this is occurring I would suggest that you run 5 thread dumps each 30 seconds apart and see what is running. Some process is consuming too much memory when it is running and is usually caused by either a bad agent or a bad bulk plan.

    Nick
    If my post was helpful please click on the Thanks! button

    MavenWire Hosting Admin
    15 years of OTM experience

    Comment


    • #3
      Re: Very unstable appserver since Linux move

      Kristof,

      In addition to the assistance that Nick is giving, I'd recommend bumping up to 2200MB for your WebLogic heap -- as a large part of the issue appears to be Full GC related. I've had a lot of luck on Linux with heaps from 2000MB to 2200MB, though larger helps sometimes run out of native memory. Nick, however, has been successful on some very large sites with heaps as large as 2400MB. I guess it depends on whether your Red Hat Linux OS is 32 or 64-bit and what else may be running on the system.

      Scaling back on your OTM threads may also help this, as the system will grind to a halt when all available heap memory is exhausted.

      Thanks!
      --Chris

      Comment


      • #4
        Re: Very unstable appserver since Linux move

        Could following weblogic settings in the conf file cause this issue?

        # Default timeout values.
        # Do not change these without understanding their purpose.
        wrapper.cpu.timeout=120
        wrapper.ping.timeout=0
        wrapper.startup.timeout=0
        wrapper.shutdown.timeout=0
        wrapper.restart.delay=5

        We changed these settings to prevent the Wrapper is going to stop/startup the appserver automatically.

        What about the following settings I'm currently testing on one of our testboxes? Any experience with ?

        wrapper.java.additional.22=-XXtlasize64k
        wrapper.java.additional.23=-XXlargeobjectlimit8k
        and
        wrapper.java.additional.4=-Xgcarallel



        # Java Additional Parameters
        wrapper.java.additional.1=-jrockit
        wrapper.java.additional.2=-Xms2048m
        wrapper.java.additional.3=-Xmx2048m
        wrapper.java.additional.4=-Xgcarallel
        wrapper.java.additional.5=-Xverbose:memory
        wrapper.java.additional.6=-Xnoclassgc
        wrapper.java.additional.7=-Dglog.home=%GLOG_HOME%/glog/config
        wrapper.java.additional.8=-Duser.home=%GLOG_HOME%/glog/config
        wrapper.java.additional.9=-Djava.io.tmpdir=%GLOG_HOME%/temp
        wrapper.java.additional.10=-Djava.awt.headless=true
        wrapper.java.additional.11=-Djava.security.auth.login.config=%GLOG_HOME%/glog/config/gc3_jaas.config
        wrapper.java.additional.12=-Dweblogic.Name=gc3-fikylx25
        wrapper.java.additional.13=-Dbea.home=/opt/otm/bea
        wrapper.java.additional.14=-Dweblogic.home=%WL_HOME%/server
        wrapper.java.additional.15=-Dweblogic.management.username=system
        wrapper.java.additional.16=-Dweblogic.management.password=%WL_PW%
        wrapper.java.additional.17=-Dweblogic.ProductionModeEnabled=true
        wrapper.java.additional.18=-Dweblogic.management.discover=false
        wrapper.java.additional.19=-Djava.security.policy=%WL_HOME%/server/lib/weblogic.policy
        wrapper.java.additional.20=-Xss256k
        wrapper.java.additional.21=-Xmanagement
        wrapper.java.additional.22=-XXtlasize64k
        wrapper.java.additional.23=-XXlargeobjectlimit8k
        Regards,

        Kristof

        Comment


        • #5
          Re: Very unstable appserver since Linux move

          Kristof,

          Looking at your settings, I have the following comments:

          wrapper.cpu.timeout=120
          CP> I understand why you changed this one -- though I hope you don't have many GCs that take longer than 60-90 seconds. JRockit is very fast, even at full GCs, so if this is the case, than either your server is swapping, your heap is too small (I'd move up to 2200MB), or you're running too much in the system at once and need to throttle it, in order to increase performance. This may not seem intuitive, but if you throttle and reduce JVM heap memory usage, JRockit wlll be able to process
          wrapper.startup.timeout=0
          CP> This should be 60 - no reason to change.
          wrapper.shutdown.timeout=0
          CP> This should be 120. If WebLogic doesn't shutdown in 120 seconds, there is an issue and JSW should kill it. This setting ensures that WebLogic is always shutdown before the next one starts up and you don't end up with hung or zombie processes.
          wrapper.java.additional.22=-XXtlasize64k
          CP> I honestly haven't had to use this one in the past. I've seen it referenced in a developer's blog in the past, but haven't had a system where it was necessary to utilize. (Henrik Stahl's Blog: June 2006 Archive)wrapper.java.additional.23=-XXlargeobjectlimit8k
          CP> I honestly haven't had to use this one in the past. I've seen it referenced in a developer's blog in the past, but haven't had a system where it was necessary to utilize. (Henrik Stahl's Blog: June 2006 Archive)wrapper.java.additional.4=-Xgcarallel
          CP> I've had much better luck with "-Xgcprio:throughput", as this allows JRockit to dynamically switch GC algorithms while running -- to ensure that the JVM adapts to the current usage and load.

          Honestly, I believe that your greatest gains would be from increasing the heap, and adjusting the workload so that the JVM heap doesn't get overwhelmed -- thus avoiding full GCs and GC memory lock issues; and thereby creating a much faster system.

          Hope this helps!

          --Chris

          Comment

          Working...
          X