Scalability with hw load balancing

[Scott T.]

Hello,
We're in the process of implementing scalability and the model we're planning has some uniqueness that I'm looking to validate (i.e. variation from the doc/diagram in webhelp). We'll be using a hardware load balancer, with two machines combining a web server and an app server, two separate machines with report servers, and a separate machine running the database server.
The load balancer would be distributing the incoming requests to the two web servers. The app servers would be configured in Active-Failover mode (a.k.a. active-passive). Both web servers would send the incoming requests to the active app server. Control of which app server is active and which is in failover mode would be handled by the OTM SCA functionality (the load balancer would not be involved). The report server machines are off to the side (each app server has one) - they may be using a Dataguard copy of the DB.
The main point I'm looking to validate is that the load balancer distributing requests to the two web servers is not going to cause problems with OTM's scalability model - which may be expecting only one web server (?). And that having the two web servers, each residing on the same physical machine as an app server, will likewise not cause problems with the OTM Active-Failover model.
Also, there is another school of thought that the load balancer should be in control of which app server is the active one. However, I think this would be outside of OTM's scalability model and would not provide the seamless failover of the user's sessions in the event of an app server failure. Thoughts on this?
Thanks and Regards,
Scott T.

[nick.polites]

Re: Scalability with hw load balancing

Scott,

You won't have any issues with the way you are setting this up. I've set scalability on about 3-4 customers with a mix of active-active and active-failover and it will work. The way you distinguish each tier instance is via the glog.properties. Make sure your load balancer is set to persistence for the cookie session otherwise you'll get bounced from one web server to the other by your LB in which you'll be asked to continuously re-login. The admin guide is very vague, unless it was recently updated so take your time when configuring it. I am almost 99% sure that what is in the admin guide for active-failover is not correct. Furthermore if you failover to the 2nd app server you need to take the first app server out of the pool otherwise you will have issues with integration. If you are planning on using 5.5 CU-3 or above I would recommend you have it set to active-active.

I am not sure if you know this but OTM does not support multiple reports instances in an active-active or active-failover mode. You’ll need to do this manually.

[chrisplough]

Re: Scalability with hw load balancing

I completely agree with Nick. The only suggestions I'd like to make is to start on OTM v5.5 CU04 RU01 or later. It is only with RU04 and above that Oracle is fully supporting SCA (scalability).

--Chris

[Scott T.]

Re: Scalability with hw load balancing

Thanks Nick and Chris.

We were under the impression that there would be more JMS overhead with active-active, and for this reason we were recommended to go with active-failover. With roughly 300 users, half of those full time, does this seem reasonable? Other factors we should be weighing in?

Nick, I'm not clear on your comment that if we failover to the 2nd app server, we would need to take the first app server out of the pool. Do you mean if we're directing traffic at the app servers via the LB? Otherwise, this would be handled by OTM SCA, wouldn't it?

I wasn't aware that multiple report instances wasn't supported. We can change our plans to use just one report instance - or can you briefly touch on the manual process you eluded to?

Lastly, we are 5.5 CU04, but haven't installed RU01. Chris - do you know if we're lacking anything pertinent to SCA by not having the RU installed?

Thanks,
Scott T.

[nick.polites]

Re: Scalability with hw load balancing

For the reports server you can get a bit creative by using an alias for the report like reports.company.com and have it pointing in your dns to reports1.company.com. When you fail over to reports2.company.com you just change your DNS to point to reports2. Keep in mind that OAS which is used for reports has a clustering ability by default so it will detect and use both reports instances if they have the same report server component name. Reports2 will need to be turned off until you need to fail over to it.

"Nick, I'm not clear on your comment that if we failover to the 2nd app server, we would need to take the first app server out of the pool. Do you mean if we're directing traffic at the app servers via the LB? Otherwise, this would be handled by OTM SCA, wouldn't it?"


SCA will fail you over to the 2nd app server but I have seen at a customer's site when they failed over to the 2nd one that some processes within weblogic, specifically agents, hung until you removed the 1st application server from the pool. This will make more sense when you configure this but I was told by development, this is the only way to truly failover the instance. It is not an automatic process but one which you will need to intervene. That is why I was suggesting active-active instead.

Keep in mind that OTM does not alert you when you have failed over to the 2nd application server so you will need to setup a 3rd party monitoring application to check this. I have integrated scripts with Nagios to do this, if you are interested we at Mavenwire can implement this for you. Please PM me if you are interested and I'll get you in contact with someone to start the process.

[chrisplough]

Re: Scalability with hw load balancing

Scott,

In addition to Nick's comments (all valid), I did want to address one question directly:

We were under the impression that there would be more JMS overhead with active-active, and for this reason we were recommended to go with active-failover. With roughly 300 users, half of those full time, does this seem reasonable? Other factors we should be weighing in?

Other factors do weight in, though active/active is often be too heavy for many customers. Only by looking at the product utilization, data configuration, failover requirements, etc. can you figure out which SCA mode is right for you. The 4 are active/active, active/passive, domain split and functional split. SCA has become very popular recently and I know of customers evaluating all 4 flavors.

Remembering your configuration, I would think that either active/passive (simply for failover), domain split (separating out business units) or functional split would be best. The latter two would give performance gains -- which is going to be important within your environment.

One final note - OTM 6.0 is supposed to improve active/active SCA -- though I will need to test it internally for performance before recommending it to anyone. Not something to bet the farm on, but something to keep in mind.

--Chris

[Scott T.]

Re: Scalability with hw load balancing

Thanks again Nick and Chris. We are headed down the path of active-failover SCA. There was mention earlier of OTM not supporting multiple Report Servers, and possibly doing something creative with the DNS, etc. What we're looking at is using the LB to distribute load between two Report Servers, no OAS clustering. Each App Srvr would communicate to 'reports.company.com', and the LB would distribute to either 'reports1.company.com' or 'reports2.company.com'. Does this seem to be a valid approach?
Regards,
Scott T.

[nick.polites]

Re: Scalability with hw load balancing

The only way this would possibly work is if you give both reports server the same Reports instance ID ie. Reports_OTM which can be done using the AddServerTarget.sh. You may run into issues scheduling reports and even running them. If you run into issues you won't be able to get Oracle support to help you as this is not supported.

[Scott T.]

Re: Scalability with hw load balancing

I've brought up the Web & App Srvrs (one of each on two Linux machines) with the glog.properties SCA definitions in place (had them up without SCA and verified they were functional). It appears I'm having jms related issues. Attempting to access from a brower I get: cause.Unknown_Service (serviceName=SecuritySessionHome)
javax.naming.NamingException: Lookup error: javax.naming.AuthenticationException: Not authorized; .
In logs I'm seeing errors such as:
==<otm-home>/oas/console.log:
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 -- OTM Startup: serverReady
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 -- OTM Startup: activating startup classes
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 : ERROR : 0
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 en : ERROR : 401209
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 en_US : ERROR : 0
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 : EXPLANATION : 0
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 en : EXPLANATION : 65584
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 en_US : EXPLANATION : 0
INFO | jvm 1 | 2008/04/24 18:41:15 | 08/04/24 18:41:15 Unknown service: jms/TopicConnectionFactory
INFO | jvm 1 | 2008/04/24 18:41:15 | javax.naming.NameNotFoundException: jms/TopicConnectionFactory not found
INFO | jvm 1 | 2008/04/24 18:41:15 |
::::
INFO | jvm 1 | 2008/04/24 18:44:11 | 2008-04-24 18:44:11.594 NOTIFICATION J2EE RMI-00003 Invalid username or password for default (guest). Authentication failed.
==<otm-home>/glog.web.log
2008-04-24 18:44:11.753 0 Error Exception cause.Unknown_Service (serviceName=TranslationSessionHome)
javax.naming.NamingException: Lookup error: javax.naming.AuthenticationException: Not authorized; nested exception is:
javax.naming.AuthenticationException: Not authorized [Root exception is javax.naming.AuthenticationException: Not authorized]
::::
2008-04-24 18:44:11.968 0 Error GLog Could not parse n/a for host and port [TP-Processor24]
2008-04-24 18:44:12.039 0 Error Exception cause.Unknown_Service (serviceName=SecuritySessionHome)
javax.naming.NamingException: Lookup error: javax.naming.AuthenticationException: Not authorized; nested exception is:
javax.naming.AuthenticationException: Not authorized [Root exception is javax.naming.AuthenticationException: Not authorized]
==<otm-home>/glog.exception.log
2008-04-24 18:41:15.947 0 Error Exception Unknown service: jms/TopicConnectionFactory
javax.naming.NameNotFoundException: jms/TopicConnectionFactory not found
==<otm-home>/glog.application.log
2008-04-24 18:41:15.946 0 Error Exception Unknown service: jms/TopicConnectionFactory
javax.naming.NameNotFoundException: jms/TopicConnectionFactory not found

Have you come across something along these lines before?

Regards,
Scott T.

[nick.polites]

Re: Scalability with hw load balancing

I haven't used OAS and performance wise it isn't the best thing for our customers either so I won't be able to help put on this error. I have seen other running into this error on the forum. I would suggest that you try searching for on the forum that error

J2EE RMI-00003 Invalid username or password for default (guest).


Nick