Announcement

Collapse
No announcement yet.

After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hangs)

Collapse
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hangs)

    Hello,

    I've just upgraded our DEV environment from 6.2.2 to 6.2.8. Installed the patch set in both Web and App server (ours is a 2 node setup - one app server, running Weblogic and the other being the Web server, runnin Tomcat / Apache. DB is a 2 node RAC running 11gR2)

    After finishing the patch, I've requested our DBA to bounce the database and after that, when I tried to start the services, Web server starts up without any issues. But App server doesn't. It just displays the message "Waiting for DB Server to finish startup" and just hangs. Looks like "$JAVA_HOME/bin/java glog.util.admin.WaitForServer <Server_name> 1521" doesn't get any response.

    Any help is greatly appreciated.

    Thanks
    Muthu
    Tags: 6.2.8, app server not starting, rac
  • #2
    Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

    Please attach your /otm/logs/weblogic/console.log.0 file.

    Nick
    If my post was helpful please click on the Thanks! button

    MavenWire Hosting Admin
    15 years of OTM experience

    Comment

    • #3
      Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

      Muthu,

      Make sure the IP Address or Hostname and port number for your database that is specified in the startit() function in the otmapp62 file is correct. Before starting the App Server the DB will be pinged via command $JAVA_HOME/bin/java glog.util.admin.WaitForServer. If the server isn't available or the IP is incorrect then the script will not proceed to start the App server.
      Last edited by itnole; March 7, 2013, 23:20.

      Comment

      • #4
        Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

        Hello Nick,

        Thanks. Uploading the console.log.0 file as a zip. FYI, since it didn't appear to start the app services, what I did was that, commenting out the line "$JAVA_HOME/bin/java glog.util.admin.WaitForServer ..." as follows, in the startit function :

        =====
        startit() {
        cd $APP_HOME
        # echo Waiting for DB Server to finish startup
        # $JAVA_HOME/bin/java glog.util.admin.WaitForServer otmgdev_otmgdev.val.vlss.local 1521
        # sleep 5
        su otm -c "./weblogic.sh start"
        }
        =====

        After that it started. console.log.0 didn't has not recorded any error.

        I modified the otmapp62 script as above at 10:40 on 03/07/13. If you look at the console.log, it only shows the log entries when it started after the above change has been made. Here're the excerpts from the logfile :

        =====
        ....
        ....
        INFO | 2013/03/07 10:31:21 | <Mar 7, 2013 10:31:21 AM EST> <Alert> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>
        INFO | 2013/03/07 10:31:22 | <Mar 7, 2013 10:31:22 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
        INFO | 2013/03/07 10:31:22 | Stopped Launched Process...
        /* My comment : There is no information reg. the hanging, here */
        INFO | 2013/03/07 10:40:41 | Starting Launched Process...
        INFO | 2013/03/07 10:40:41 | java version "1.6.0_20"
        ....
        ....
        =====

        Thanks and regds.
        Muthu
        Attached Files
        Last edited by muthu.kumar; March 12, 2013, 16:18.

        Comment

        • #5
          Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

          The wait for server will sit there until it sees that the server and port are listening. In your case it can't find the database otmgdev_otmgdev.val.vlss.local 1521. So either you have a network issue or a DNS issue.

          There should be more information in your console.log above what you posted. I need to see the entire log otherwise I can't be of much help.

          Nick
          If my post was helpful please click on the Thanks! button

          MavenWire Hosting Admin
          15 years of OTM experience

          Comment

          • #6
            Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

            Also your database DNS doesn't look correct. You'll need to run a recursive grep for otmgdev_otmgdev.val.vlss.local and update it to your database DNS name.
            If my post was helpful please click on the Thanks! button

            MavenWire Hosting Admin
            15 years of OTM experience

            Comment

            • #7
              Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

              Thanks again Nick, for the quick update.

              To both you and itnole :

              We didn't have this issue prior to the upgrade. Our database is a 2 node RAC. I'm also a little confused about what should be the settings.

              Here're the contents of the tnsnames.ora and the glog.properties files :

              =====

              [[email protected] weblogic]$ echo $TWO_TASK
              otmgdev

              =====

              tnsnames.ora
              -------------

              OTMGDEV =
              (DESCRIPTION =
              (ADDRESS_LIST =
              (ADDRESS = (PROTOCOL = TCP)(HOST = otmgdev_otmgdev.val.vlss.local)(PORT=1521))
              )
              (CONNECT_DATA =
              (SERVICE_NAME = otmgdev.val.vlss.local)
              (SERVER = DEDICATED)
              )
              )

              =====

              Database related parameters in glog.properties :
              ----------------------------------------------

              glog.database.dbaOnly.t2client.connectionURL=jdbcracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALAN CE=YES)(ADDRESS=(PROTOCOL=TCP)(HOST=otmgdev_otmgde v1.val.vlss.local)(PORT=1521))(ADDRESS=(PROTOCOL=T CP)(HOST=otmgdev_otmgdev2.val.vlss.local)(PORT=152 1)))(CONNECT_DATA=(SERVICE_NAME=otmgdev)))
              # glog.database.dbaOnly.t2client.connectionURL=jdbcracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALAN CE=YES)(ADDRESS=(PROTOCOL=TCP)(HOST=otmgdev_otmgde v.val.vlss.local)(PORT=1521)))(CONNECT_DATA=(SERVI CE_NAME=otmgdev_otmgdev)))
              # glog.database.dbaOnly.t2client.connectionURL=jdbcracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(P ROTOCOL=TCP)(HOST=otmgdev_otmgdev.val.vlss.local)( PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=otmgdev.va l.vlss.local)(SERVER=DEDICATED)))
              glog.database.dbathin.t2client.connectionURL=$glog .database.dbaOnly.t2client.connectionURL$
              glog.database.DBA.t2client.connectionURL=$glog.dat abase.dbaOnly.t2client.connectionURL$
              glog.database.DBADirect.t2client.connectionURL=$gl og.database.dbaOnly.t2client.connectionURL$
              glog.database.migration.t2client.connectionURL=$gl og.database.dbaOnly.t2client.connectionURL$
              glog.database.dbathin.t2client.connectionURL=$glog .database.dbaOnly.t2client.connectionURL$
              glog.database.dbaglogowner.t2client.connectionURL= $glog.database.dbaOnly.t2client.connectionURL$
              glog.database.dbareportowner.t2client.connectionUR L=$glog.database.dbaOnly.t2client.connectionURL$
              dbserver=otmgdev_otmgdev1.val.vlss.local,otmgdev_o tmgdev2.val.vlss.local
              # dbserver=otmgdev_otmgdev.val.vlss.local
              glog.database.sid=otmgdev

              =====

              I could ping the otmgdev_otmgdev.val.vlss.local as follows :

              =====

              [[email protected] config]$ ping otmgdev_otmgdev.val.vlss.local
              PING wgpor005-scan.val.vlss.local (10.0.5.124) 56(84) bytes of data.
              64 bytes from 10.0.5.124: icmp_seq=1 ttl=64 time=1.37 ms
              64 bytes from 10.0.5.124: icmp_seq=2 ttl=64 time=0.170 ms
              64 bytes from 10.0.5.124: icmp_seq=3 ttl=64 time=0.244 ms

              --- wgpor005-scan.val.vlss.local ping statistics ---
              3 packets transmitted, 3 received, 0% packet loss, time 2000ms
              rtt min/avg/max/mdev = 0.170/0.597/1.378/0.553 ms
              [[email protected] config]$

              =====

              Thanks and regds.
              Muthu

              Comment

              • #8
                Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                Can you ping otmgdev_otmgdev.val.vlss.local?
                If my post was helpful please click on the Thanks! button

                MavenWire Hosting Admin
                15 years of OTM experience

                Comment

                • #9
                  Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                  Yes Nick, I could ping otmgdev_otmgdev.val.vlss.local.

                  Thanks
                  Muthu

                  Comment

                  • #10
                    Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                    OK please attach the console.log.0 file so I can have a look at what the issue may be.
                    If my post was helpful please click on the Thanks! button

                    MavenWire Hosting Admin
                    15 years of OTM experience

                    Comment

                    • #11
                      Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                      Thanks Nick. I attached it already as a zip file.

                      Comment

                      • #12
                        Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                        Your app started without any problems so that doesn't seem to be the issue. Can you attach your Tomcat console.log?
                        If my post was helpful please click on the Thanks! button

                        MavenWire Hosting Admin
                        15 years of OTM experience

                        Comment

                        • #13
                          Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                          Yes Nick, since it was hanging, I commented those lines from otmapp62 and started the app server. It started and running fine since then. We're doing the testing of the 6.2.8 upgrade and the development team is good so far.

                          My only concern is that I shouldn't have to change the OTM supplied scripts like this. And from tomorrow, we're going to start cloning our prod into QA environment to upgrade QA.

                          I thinks this is something to do with the configuration of the DB related parameters only, as the DB is RAC. Earlier, we encountered an "ORA-12505 TNS:listener does not currently know of SID" error while running update_password.sh to reset the password for DBA.ADMIN, for which I've an SR open with Oracle Support. Then, I had to temporarily update the DB related parameters of glog.properties, pointing to one node and run the same which ran successfully.

                          Attaching the tomcat console.log file for your review.

                          Thanks and regds.
                          Muthu
                          Attached Files

                          Comment

                          • #14
                            Re: After 6.2.8 upgrade, App server doesn't start (glog.util.admin.WaitForServer hang

                            There is a section that needs to be setup differently for RAC. I would suggest having a look there to see if something changed with 6.2.8. Be sure to download off of Oracle's site the latest version of the Admin guide.

                            Nick
                            If my post was helpful please click on the Thanks! button

                            MavenWire Hosting Admin
                            15 years of OTM experience

                            Comment

                            Previous template Next
                            Working...
                            X