Announcement

Collapse
No announcement yet.

Performance issue with webservers in OTM 5.5

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Performance issue with webservers in OTM 5.5

    Hi All,

    We recently face performance issues with webservers.

    The CPU usage is very high say 400 % in some of the sebservers which is affecting the OTM performance.

    Can anyone tell us what all may be the reasons behind this high cpu usage.

    Thanks in advance

    Regards
    Dhivya

  • #2
    Re: Performance issue with webservers in OTM 5.5

    Try running thread dumps while the issue is occurring and post the console.log here.

    To run a thread dump in Linux find the PID for tomcat (ps -ef | grep java) and then run a kill -3 PID. Run this 6 times every 30 seconds.

    Nick
    If my post was helpful please click on the Thanks! button

    MavenWire Hosting Admin
    15 years of OTM experience

    Comment


    • #3
      Re: Performance issue with webservers in OTM 5.5

      Hi Nick,

      Thanks sure i will upload the thread dumps that you asked for.

      Thanks so much

      Regards,
      Dhivya

      Comment


      • #4
        Re: Performance issue with webservers in OTM 5.5

        Hi Dhivya,

        Depending on what CU you're on this MOS document may or may not apply to you. I know we applied these updates some time ago for a similar issue.

        Updating Third Party Jar Files For Improved UI Performance [ID 471363.1]

        Reads in part

        The following steps can be used to determine if the new .jar files are installed on your OTM instance. This check should be done on the OTM Web and Application servers.

        cd < OTM Home >/glog/gc3webapp/WEB-INF/lib
        md5sum standard.jar
        cd 3rdparty
        md5sum standard.jar
        md5sum jaxen-full.jar

        The md5sum results should be "fcd8b968265683bafae757f94af19548" for the standard.jar The md5sum results should be "261d1aa59865842ecc32b3848b0c6538" for the jaxen-full.jar

        If your md5sum results do not match the values mentioned previously the new .jar files have not been installed on your OTM instance and you should follow the download and installation instructions provided below.
        Thanks,
        Alan

        Comment


        • #5
          Re: Performance issue with webservers in OTM 5.5

          Hi Nick,

          Please find the attached thread dumps collected from the web serverthread_dumps.txt when the issue occurred.

          the issue occurred early in the morning around 5:00 to 6:00 IST.

          Thanks
          Dhivya


          Originally posted by dhivya.enjoy View Post
          Hi Nick,

          Thanks sure i will upload the thread dumps that you asked for.

          Thanks so much

          Regards,
          Dhivya

          Comment


          • #6
            Re: Performance issue with webservers in OTM 5.5

            Hi Alan,

            Thanks for the update.

            Let me check and come back to you if this is required for us.

            Regards,
            Dhivya

            Originally posted by acuartero View Post
            Hi Dhivya,

            Depending on what CU you're on this MOS document may or may not apply to you. I know we applied these updates some time ago for a similar issue.

            Updating Third Party Jar Files For Improved UI Performance [ID 471363.1]

            Reads in part

            Thanks,
            Alan

            Comment


            • #7
              Re: Performance issue with webservers in OTM 5.5

              Hi Nick,

              Was the thread dumps provided is sufficient enough to check?

              Regards,
              Dhivya

              Comment


              • #8
                Re: Performance issue with webservers in OTM 5.5

                Hi Dhivya,

                I guess we're assuming that it's the java process (tomcat) that is consuming the CPU on the web server but did you actually confirm this by running top? Have you collected any sar stats, if not it might be worth doing that too (maybe there is an impact in another subsystem such as heavy network traffic). You might also want to look in $OTM_HOME/temp/upload to see if there were any especially large uploads at the time that the CPU ramped up. It's also worth looking in glog.web.log for any unusual activity. Based on your console log, memory does not seem to be the issue the last full GC I see in there is clearing down to 415MB. Sorry, the thread dumps you sent don't mean much to me (I am not a java developer) sometimes I can spot a stuck thread but most often I just send them off to Oracle for analysis. Maybe Nick will come back with more info on that.

                Sometimes web servers can go nuts because a user will attempt to bring up an especially large object for editing (like a shipment with hundreds of ship units). You may want to check to see if anyone is having a problem editing specific objects in the UI. In our case we had several cases like this but usually the issue would manifest itself as high memory utilization not CPU. There are properties that can be added to limit the number of child objects that can be edited (so called Query limits). This is what we have in place and what seems to be reasonable for most cases we have

                Code:
                #### Query Limits
                glog.query.limit.glog.server.query.orderbase.ObLineQuery=150
                glog.query.limit.glog.server.query.orderbase.ObShipUnitQuery=150
                glog.query.limit.glog.server.query.order.OrderReleaseLineQuery=150
                glog.query.limit.glog.server.query.order.ShipUnitQuery=150
                glog.query.limit.glog.server.query.order.ShipUnitLineQuery=150
                glog.query.limit.glog.server.query.shipment.ShipmentStopDQuery=150
                glog.query.limit.glog.server.query.shipment.SShipUnitQuery=150
                glog.query.limit.glog.server.query.shipment.SShipUnitLineQuery=150
                glog.query.limit.glog.server.query.invoice.InvoiceLineitemQuery=150
                glog.query.limit.glog.server.query.invoice.VoucherInvoiceLineitemJoinQuery=150
                I think we have also seen where the Map Related Shipments function can cause web server issues due to the graphical nature of this action using large amounts of memory. Unfortunately I have never seen this issue first hand because we stopped using it before I started here but you may want to check to see if anyone is using that function.

                Thanks,
                Alan

                Comment


                • #9
                  Re: Performance issue with webservers in OTM 5.5

                  Are your users running lots of CSV extracts? I am seeing in the thead dumps multiple lines

                  INFO | jvm 1 | 2012/04/19 01:28:16 | at glog/integration/servlet/CSVUtilServletHelper.process(CSVUtilServletHelper. java:172)
                  INFO | jvm 1 | 2012/04/19 01:28:16 | at glog/integration/servlet/BatchCSVUtilServlet.doGet(BatchCSVUtilServlet.java :62)

                  Besides that I see the WMservlet which is integration.

                  I would ask your users to stop the CSV Util extract and see if the issue goes away.

                  Nick
                  If my post was helpful please click on the Thanks! button

                  MavenWire Hosting Admin
                  15 years of OTM experience

                  Comment


                  • #10
                    Re: Performance issue with webservers in OTM 5.5

                    Hi Nick,

                    On a totally un related topic.. i am looking for a Util servlet i can use to post XML into OTM..

                    We are using "glog.integration.servlet.BatchCSVUtilServlet" servlet for loading CSV files into OTM
                    do we have a servlet that could be used to load an XML files into OTM?

                    Regards,
                    Ranga

                    Comment


                    • #11
                      Re: Performance issue with webservers in OTM 5.5

                      Hi Ranga,

                      From the Integration Guide, you can use "glog.integration.servlet.WMServlet" to load an XML to OTM.

                      Regards,
                      Nando.

                      Comment


                      • #12
                        Re: Performance issue with webservers in OTM 5.5

                        Nando is right - post your XML files to WMServlet. This is the primary integration servlet and is still how most companies post to OTM.

                        --Chris

                        Comment

                        Working...
                        X