Symptoms
OA CP is down, the following message is logged to /var/log/poa.log:
2016-03-01T01:14:33.458330+01:00 osscore.domain.tld : DBG [UI:pre-login:783588:1456782861458 RequestProcessor-17596 pau ]: com.parallels.pa.service.otsjta.OTSTransactionImpl commit failedjavax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commit(TransactionImple.java:211) ~[jbossjts-jacorb-4.17.15.Final-redhat-4.jar!/:4.17.15.Final-redhat-4]
at com.parallels.pa.service.otsjta.OTSTransactionImpl.commit(OTSTransactionImpl.java:244) ~[poakernel-ejb.jar:na]
at org.omg.CosTransactions.TerminatorPOATie.commit(TerminatorPOATie.java:59) [jbossjts-jacorb-4.17.15.Final-redhat-4.jar!/:4.17.15.Final-redhat-4]
at org.omg.CosTransactions.TerminatorPOA._invoke(TerminatorPOA.java:51) [jbossjts-jacorb-4.17.15.Final-redhat-4.jar!/:4.17.15.Final-redhat-4]
2016-03-01T01:14:33.458352+01:00 osscore.domain.tld : at org.jacorb.poa.RequestProcessor.invokeOperation(RequestProcessor.java:306) [jacorb-2.3.2.redhat-5.jar!/:2.3.2.redhat-5]
at org.jacorb.poa.RequestProcessor.process(RequestProcessor.java:626) [jacorb-2.3.2.redhat-5.jar!/:2.3.2.redhat-5]
at org.jacorb.poa.RequestProcessor.run(RequestProcessor.java:769) [jacorb-2.3.2.redhat-5.jar!/:2.3.2.redhat-5]
java.lang.OutOfMemoryError: GC overhead limit exceeded
There was no global memory shortage, so something ate up the memory inside PAU.
Cause
Memory leak in PAU daemon.
Resolution
In order to trace down similar issues, it is needed to append -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/pau
options to the PAU init script /etc/init.d/pau
at the following spot:
JAVA_MEMORY_OPTS="-Xms1303m -Xmx1303m -XX:MaxPermSize=256m"
so that it turns to:
JAVA_MEMORY_OPTS="-Xms1303m -Xmx1303m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/pau"