Application Server out of Memory - glog.webserver.mediator.MediatorDiagServlet

[james_foran]

Hello All,

Had a situation today where the OTM Application SErver (OTM 5.0) required a restart as it was performing Full GC every 30 seconds, and was unable to free any memory.

The thread utilization was minimal. Further investigation revealed that the there was a huge amount of events stuck in glog.webserver.mediator.MediatorDiagServlet..

I have not used this before.

We had one Group Type "CustomShipmentEvent" which has 175083 items loaded it it.

We have a recurring process which was executing every minute that was responsible for the large number of Items, however, why the items were not being processed by the Application is a mystery.

Following a restart, everything looks fine again. These custom events were building up for an entire week.

We are working with Mavinwire on this issue, but any assistance from others would ge greatly appreciated.

Thanks and regards,

James Foran
Toll Solutions

[chrisplough]

Re: Application Server out of Memory - glog.webserver.mediator.MediatorDiagServlet

James,

I know we've spoken about this, but I hope this information helps others here on the Forum:

The mediator is a subsystem wrote to block dependent workflow topics from executing in parallel. It's used for a host of asynchronous coordination tasks,blocking:

• ordered transactions within a transmission (process-in-sequence transactions block each other)
• integration updates on the same business object
• agent actions on the same business object
• agents on the same business object (if the agent has the Lock flag set)

When a topic is published, it first goes to the mediator. It's assigned to one or more groups, denoting the type of activities in wants to block. E.g. an order mod can be in a transmission group to ensure ordering, and an order group to block multiple mods on the same order. Each topic is also assigned a unique sequence to determine its relative priority in the group. Whenever a topic completes, it frees up its mediator groups to look for the next topic with no group blocks. This topic is then published to the event queues.

If there is a lot of mediator blocking, then, you can end up with a slow, contentious system even though there are many free event queue threads. A single long-running query can block up transactions in the mediator.

Prior to 5.5CU2, the mediator was used primarily for transactions. Blocking UI activity was done via object locks, coordinated against the mediator. As of 5.5CU2, we added agent actions to the mix. Every agent action now goes through the mediator, blocked by the business object it works on. This ensures that two agents running against the same shipment won't execute agent actions simultaneously. Going one step forward, the users can set the Lock flag on the agent which blocks the entire agent in the mediator. If one agent is running against a shipment, another agent acting on the same shipment will wait until the first completes.

The MediatorDiagServlet shows the active topics in each mediated group, along with all the topics waiting for the group to be available. If the mediator is backing up, you generally have two much coordinated activity (e.g. unnecessary serial transactions, too many updates to the same shipment). Chances are there are too many agents or actions firing off for the same shipment.

So - it seems that we need to find the root-cause of blocking and resolve - rather than trying to utilize the glog.workflow.mediator.customShipmentEvent.mediati onEnabled property.

--Chris Plough

In addition, there is a property to help control this behavior:

glog.workflow.mediator.customShipmentEvent.mediationEnabled=[true|false]

This is related to issue 55722.5.0, which was included in the May 2006 SR of v5.0:

Issue: Mediation blocks agent execution
Resolution: added a new workflow property (glog.workflow.mediator.customShipmentEvent.mediati onEnabled). The default is true. To turn off mediation on custom shipment events, set the property to false.

--Chris

[james_foran]

Re: Application Server out of Memory - glog.webserver.mediator.MediatorDiagServlet

Issue Description:

1. When a Custom Shipment Event is raised, it is first passed to the Mediator on the Application Server. The mediator ensures that two similar actions cannot occur on a single object (shipment) at the same time.
2. The Mediator then triggers that event, and holds onto the event until the process is finished.
b. If within the Agent, or agents triggered by the custom event, we raise another custom event, and do NOT create a new process, the second event is passed to the Mediator. This event never executes, as it is waiting for the first event to finish


These tasks will never complete. In the case of the Carrier Notified agents, we continue to raise new events every minute. This resulted in the restart last Tuesday, because we ended up with 170,000 Events waiting to process for about 10 IMTS shipments.

Resolution

When raising a new event from within an agent, ensure that Create New Process is selected.