Announcement

Collapse
No announcement yet.

"java.net.ConnectException: Connection refused; No available router to destination"

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • "java.net.ConnectException: Connection refused; No available router to destination"

    Hello,

    One of the app servers in our QA environment that's configured to process bulk plans is not shutting down. We've 5 app servers and 2 web servers. Among the 5, 2 are configured to be bulk plan servers to process only bulk plans and the other is configured to be a standby. Rest of the 2 app servers do the non-bulk plan activities.

    I've scheduled a nightly bounce of the servers. Every night, but for one of the bulk plan app servers, all the other 6 are properly bounced. I'm getting the following error in the shutdown.log.0 :

    *****
    Failed to connect to t3://valwgpotb002vm.val.vlss.local:7001: Destination unreachable; nested exception is:
    java.net.ConnectException: Connection refused; No available router to destination
    *****

    I can successfully ping and telnet and I've no issues. In the console.log.0, I see a lot of out of memory errors as follows :

    *****
    INFO | 2013/03/26 08:10:10 | [ERROR][memory ] Throwing OutOfMemory: getNewTla
    INFO | 2013/03/26 08:10:10 | [ERROR][memory ] Throwing OutOfMemory: [C, len 9, size 40B
    INFO | 2013/03/26 08:10:10 | [ERROR][memory ] Throwing OutOfMemory: getNewTla
    *****

    And when the shutdown command was issued, here're the error messages logged :

    *****
    WARNING | 2013/03/26 22:02:58 | Launched Process Forcibly Killed.
    WARNING | 2013/03/27 22:03:00 | Launched Process Forcibly Killed.
    WARNING | 2013/03/28 22:02:58 | Launched Process Forcibly Killed.
    WARNING | 2013/03/29 22:03:00 | Launched Process Forcibly Killed.
    WARNING | 2013/03/30 22:02:58 | Launched Process Forcibly Killed.
    WARNING | 2013/03/31 22:03:00 | Launched Process Forcibly Killed.
    WARNING | 2013/04/01 22:03:00 | Launched Process Forcibly Killed.
    *****

    This has been going on only for the last 2 weeks, since we upgraded our environment to 6.2.8 from 6.2.2. Since the server couldn't be shutdown, the JVM is up and running and the command to shutdown weblogic server "weblogic.sh stop" hangs.

    Any idea what could be the problem ?

  • #2
    Re: "java.net.ConnectException: Connection refused; No available router to destinatio

    If you are hitting OOM, you will have to kill the process. It will just sit there otherwise.

    How much memory do you have allocated for this app server?

    Nick
    If my post was helpful please click on the Thanks! button

    MavenWire Hosting Admin
    15 years of OTM experience

    Comment


    • #3
      Re: "java.net.ConnectException: Connection refused; No available router to destinatio

      I would also recommend you add the following to your weblogic.conf under the JVM parameters:

      jvm.arg=-XX:+HeapDumpOnOutOfMemoryError
      jvm.arg=-XX:HeapDumpPath=%GLOG_HOME%/logs/weblogic


      The next time you get an OOM, it wil dump the heap into your OTM/logs/weblogic directory. You can then provide this to OTM support and they can examine it for what may be causing the OOM. These two parameters will take after the next restart of OTM.

      Nick
      If my post was helpful please click on the Thanks! button

      MavenWire Hosting Admin
      15 years of OTM experience

      Comment


      • #4
        Re: "java.net.ConnectException: Connection refused; No available router to destinatio

        Great, thanks Nick. You're right about the process just hanging in there. I killed them. I've allocated 1.6 Gb to the JVM in all the app servers.

        I'll try those 2 parameters you've mentioned.

        Regds.
        Muthu

        Comment


        • #5
          Re: "java.net.ConnectException: Connection refused; No available router to destinatio

          Usually the minimum we use for 6.X is 4GB in test/dev and 8GB in prod. It might be worth it to increase it to 4096 and see if that helps.

          Nick
          If my post was helpful please click on the Thanks! button

          MavenWire Hosting Admin
          15 years of OTM experience

          Comment


          • #6
            Re: "java.net.ConnectException: Connection refused; No available router to destinatio

            Also, any idea what could cause the " java.net.ConnectException: Connection refused; No available router to destination" error ?

            Thanks
            Muthu

            Comment


            • #7
              Re: "java.net.ConnectException: Connection refused; No available router to destinatio

              If this was from an OOM and trying to shutdown Weblogic, it wouldn't respond so you would get that error.
              If my post was helpful please click on the Thanks! button

              MavenWire Hosting Admin
              15 years of OTM experience

              Comment

              Working...
              X