Advanced Queue

[krishna.kk]

Hi:

Did anyone worked on Advanced Queue of OTM? If so, Please send me a sample program to connect to Advanced queue using Java. Any sort of documentation regarding AQ is welcomed.

-KK

[chrisplough]

Re: Advanced Queue

kishore,

I just found this note and wanted to post the information that I had sent over to you, so that it will hopefully help the community -- sorry that you are seeing this twice

I don't have any Java code around OAQ, unfortunately, and have only seen it used at a couple of clients (Coca-Cola and UPM). There is very little documentation around this, but it does work reliably. I've worked primarily on the performance side - optimizing OAQ and OTM threads in order to increase the integration throughput.

However, I work with one of the engineers who designed the Coca-Cola OAQ integration, but he's usually working and isn't an active member of the forums.

I'll try to get more details from him and post them here.

Thanks,
Chris

[chrisplough]

Re: Advanced Queue

Vignaesh,

Can you give me some more information as to the details your looking for from an implementation sequence? Are you looking for information on how to program your integration for the queues, or simply how using OAQ differs from standard integration development?

If your question is on the latter, then it doesn't differ significantly. Rather than posting and receiving messages from OTM via the WMServlet servlet (that sounds redundant!), you publish and subscribe to queues within Oracle DB. So the true difference in your integration development is simply the programmatic interface, though all of your XMLs and datafile translations will remain the same. In order to parallelize your integration development, you can also utilize the standard OTM integration servlet via http posts, to test your XML translations, while another team perfects the OAQ interfaces -- then bring the two together for the final solution.

Thanks!
--Chris

[otmuser]

Re: Advanced Queue

Chris,

Can you also let me know the process of implementing OAQ. Also please let me know few performance tuning tips for integration.I have these 2 reks at my new client and I need this info ASAP.Appreciate if you can help me here.

I have aked the admin to increase the JVM memory for App to 4GB from default 1600 MB and increased the integration threads in Event Diag Servlet to 3 from 1.

thanks

[chrisplough]

Re: Advanced Queue

As for the tuning, I would definitely us a larger JVM. We usually use 2.4GB for OTM v5.5 and 4-8GB for OTM v6.x. Increasing the integration threads is a good step, though you'll likely need to monitor and increase others also (AgentIntegration, Execution, etc). Monitor via EventDiag and tune as you need.

--Chris

[otmuser]

Re: Advanced Queue

Thanks Chris.

I actually increased the threads for integration,integrationoutbound,agentUtility and the performance still went down from the last run.

I also increased the shipment bean count to 3000 from default 2000 in the bean cache servlet.

Can you please let me know whats the criteria to increase these threads and bean capacity?

[chrisplough]

Re: Advanced Queue

I watch the average wait time and if it gets to high (too high as determined by the customer's performance requirements), then I add threads. Each time I add threads, I look at all of the queues in the Event Queue servlet, to make sure that change doesn't cause other queues to backup. Repeat the process until you either get the performance you need or notice that adding threads decreases performance.

There's more involved to it (lots of experience over the years), but that will get you started. I'm working on some podcasts/training to take this further, but it'll be another couple of months before they're due out.

--Chris

[otmuser]

Re: Advanced Queue

Thanks Chris.

I increased the threads and along with the Shipment Bean Capacity and the performance was awful than the previous round.Then i unloaded all the beans and bounced the servers and the performance improved by 20%.

My testing involves how many transmission reports are sent per hour for the inbound load of transactions coming from a external source(integration)

i see a lot of throughput on the integration,integrationoutbound,processsweeper,tra nsport-http..do i have to increase these thread counts since the throughput is more or ?

Please advise

[chrisplough]

Re: Advanced Queue

You will definitely have to increase other thread groups if the integration throughput is high. I couldn't tell you exactly which ones without monitoring the OS performance, EventDiag, MediatorDiag and the ObjectLockDiag servlets.

Tuning OTM is an interactive process and can take a lot of iterative testing. Run a test, increase the thread groups that you're noticing long average wait times, then run the test again. You should see the system's CPU usage increase and process times decrease. As soon as you stop seeing process times decrease, you've found the optimal thread settings.

--Chris

[otmuser]

Re: Advanced Queue

Thanks Chris,

So we have to increase the other threads where there is low throughput.Some threads have 0 throughput since there is no activity there .

If a thread is waiting for so long does it mean its waiting for the related action to take place.I thought if eg: if integrationOutbound has 2 threads and one thread is waiting for 2 hrs and other thread is Active, does it mean only one thread is working and I can kill the other one.How can i find out if both the threads are working , based on the timestamp or ?

What do we monitor in the Mediator and object lock servlets and what steps do we take there?

thanks
otmuser

[chrisplough]

Re: Advanced Queue

Waiting refers to how long a process waits in the backlog queue before it's picked up by an available thread. That's why it's an important performance factor.

The Object Lock servlet shows object locks (both in-memory and database -- though you should still monitor DB locks via Enterprise Manager or similar tools). Mediator is harder to explain - that may have to wait till I put up more performance-related podcasts (just due to time).

--Chris

[otmuser]

Re: Advanced Queue

So if an object has 2 threads and one of the threads is waiting for 2 days.can we go ahead and kill it.

Whats the criteria to kill a thread?

I increased the max DB Connection pool size to 250 from 150 and left the min. value at 100.This degraded performance.....?

If there are lot of Shipments and Shipment Statuses coming in via integration which threads do we increase...ShipmentEvents and ?

thanks
otmuser

[chrisplough]

Re: Advanced Queue

I wouldn't kill a thread unless you are absolutely sure that some process is stuck and won't complete. The EventDiag screen doesn't update in real-time, so you may be killing a process you didn't intend to when killing a thread. If a thread is waiting, then that just means that it's not running anything at the moment. It doesn't mean that it's stuck.

Increasing the DB Connection Pool shouldn't decrease performance, unless your DB server is limited on memory. Otherwise, it's just more connections that it holds open and doesn't really slow anything down.

Rather than tell you which threads to increase without knowledge of the system, I recommend monitoring EventDiag and watching the average "Wait Time". That's the key performance indicator - more so that the backlog or throughput measurements. Those are the queues that are bottlenecking performance.

--Chris

[otmuser]

Re: Advanced Queue

Thanks Chris.

So let me give you an example here :

I increased threads to 3 from 1 on the Integration Queue and when the load is sent i see all the threads are waiting and one of them is Active sometimes.So what does this mean, it doesnt require 3 threads and the default thread of 1 is enough.How do I know all the threads are working ? When all of them are in Active state?

I have increased all the thread counts where there was high throughput but after that the performance actually decreased.

And our instance has 16GB of memory and we are using 4GB for App server and 2GB for the web server. Is this enough ? I heard that a 64 bit App server cannot take more than 4GB . Is this true ?

Right now the zone that my instance is , is using 54% of the available 16GB and the rest of the memory is used as swap memory.Can you tell me what should be the ideal %s here.

thanks,
otmuser

[JayVal]

Re: Advanced Queue

As for the tuning, I would definitely us a larger JVM. We usually use 2.4GB for OTM v5.5 and 4-8GB for OTM v6.x. Increasing the integration threads is a good step, though you'll likely need to monitor and increase others also (AgentIntegration, Execution, etc). Monitor via EventDiag and tune as you need.

--Chris

Hi Chris,

I am trying to understand the different columns in the EventDiag servlet. Can you please point me to any reference material for this ?

Thanks
J