Symptoms

After Odin Automation upgrade to version 7.3/7.4, web sockets do not work properly.

The following symptoms can be observed:

  1. CentOS/CloudLinux/RedHat 6 based UI servers are currently installed on the platform;

  2. A lot of "Session expired" messages in the /var/log/pa/pui/pui.log on the UI servers (if APS Booster is enabled) or in /var/log/pa/core.log on the OSS Core node:

    # grep -ic "session expired" core.log
109884

22:10:08.995 [default task-66] DEBUG c.p.p.c.c.a.APS2ProxyServletResources - Session expired
com.plesk.p2.security.SessionExpiredException: No such session
at com.plesk.p2.security.SecurityManager.get(SecurityManager.java:365)
at com.plesk.p2.security.SecurityManager.get(SecurityManager.java:351)
at com.plesk.p2.security.SecurityManager.checkOperation(SecurityManager.java:330)
at com.plesk.p2.cp.core.aps.APS2ProxyServlet.windowSessionFromRequest(APS2ProxyServlet.java:410)
at com.plesk.p2.cp.core.aps.APS2ProxyServletResources.service(APS2ProxyServletResources.java:87)

    There is also a large amount of messages of the following kind:

    Mar  5 22:04:12.612 : DBG [aps-service:1412934 aps-service-task-157:662 pau]: c.p.p.tracer exit by exception: com.parallels.pa.service.connectivity.ejb.EndpointExecutorBean.call com.parallels.pa.service.aps.api.APSEndpointExecutionException: unknown channel: 75085084-34b7-4d12-ac67-b83c38b61ea8
    at com.parallels.pa.service.connectivity.ejb.EndpointExecutorBean.throwResponseException(EndpointExecutorBean.java:394)
    at com.parallels.pa.service.connectivity.ejb.EndpointExecutorBean.sendRequestToApplication(EndpointExecutorBean.java:246)

  3. In attempt to open UX1 for Customers, in browser console log 404 Not Found error appears for the following GET:

    GET https://cp.example.com/ws/notifications/channel   404 Not Found

  4. Check the number of notifications in JBoss:

    [root@OSSCORE ~]# sh /usr/local/pem/wildfly-11.0.0.Final/bin/jboss-cli.sh -c "/subsystem=messaging-activemq/server=default/:read-children-resources(child-type=jms-topic, include-runtime=true)"
{
"outcome" => "success",
"result" => {"com.parallels.jms.topic.Notifications" => {
"delivering-count" => 0,
"durable-message-count" => 13543,
"durable-subscription-count" => 409,
"entries" => ["java:jboss/jms/topic/Notifications"],
"legacy-entries" => undefined,
"message-count" => 13543L,
"messages-added" => 13958L,
"non-durable-message-count" => 0,
"non-durable-subscription-count" => 0,
"subscription-count" => 409,
"temporary" => false,
"topic-address" => "jms.topic.com.parallels.jms.topic.Notifications"
}}
}

    Persistent large numbers in message counts indicate the issue.

  5. In case of User sessision timeout is set to more then 1 minute, UX1 session lasts infinite long.

Cause

UX1 onScreen notifications make use of the modern web-socket based protocol to deliver messages to end customers. For a RHEL6-based UI/Branding node, there is a RHEL issue related to malfunctioning mod_rewrite module that is unable to pass such requests as a proxy, reference:

mod_rewrite unable to proxy websocket URIs

The issue has no solution for RHEL6.

As a result of the issue, the onScreen notifications mechanism gets switched to long polling that results in eternal handling of browser sessions, until the user browser is closed. Such sessions generate a large amount of messages in the logs.

As a side issue, in case of lost target channels on OSS Core node due to services restart or any other disruption, the notifications may get stuck, leading to the service instability, high CPU usage and CP outages.

Resolution

The issue was fixed in the latest version of Odin Autotomation platform. Please contact your Technical Account Manager or Pooled Technical Associates(pta@odin.com) to schedule Odin Automation upgrade.

In order to apply workaround and drop long-polling notifications, contact Odin Technical Support.

Internal content

Link on internal Article