Symptoms

We tried to restart pa-agent and pau services on osscore as per

https://cloudblue.freshdesk.com/support/solutions/articles/44001890051-how-to-restart-cloudblue-commerce-system-services-ui-management-node-agents


However pa-agent fails to start now:

[root@core ~]# service pa-agent start
Starting pa-agent (via systemctl):  Job for pa-agent.service failed because the control process exited with error code. See "systemctl status pa-agent.service" and "journalctl -xe" for details.
                                                           [FAILED]
[root@core ~]# systemctl status pa-agent.service
● pa-agent.service - SYSV: Starts and stops the PA service
   Loaded: loaded (/etc/rc.d/init.d/pa-agent; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2021-02-04 05:55:28 CST; 55s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 27821 ExecStop=/etc/rc.d/init.d/pa-agent stop (code=exited, status=0/SUCCESS)
  Process: 30901 ExecStart=/etc/rc.d/init.d/pa-agent start (code=exited, status=1/FAILURE)
 Main PID: 7917 (code=exited, status=0/SUCCESS)
[root@core ~]#

In /var/log/messages we see:

Feb  4 05:55:28.164 pa-agent[30901]: Starting pa-agent: SCMonitor::startAllSC failed: Failed to list active SCs. Is service pau running?
Feb  4 05:55:28.644 pa-agent[30901]: start failed

Cause

pau service fails to start properly. 


Resolution

Search for errors/warning messages in /var/log/pa/console.log


Example warning:

210204 05:55:58,252 WARNING [javax.enterprise.resource.corba._DEFAULT_.rpc.transport] (remoting task-1) "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 192.168.222.12; port: 8355": org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No

In the above example we see that port/socket 192.168.222.12:8355 is busy and the service cannot bind to it.


The solution then will be to stop both of the services and check the socket with this command:

# netstat -plant | grep 192.168.222.12:8355

If pa-agent and pau services are stopped, the output should be empty.


If the output is not empty, eliminate the process that is holding the socket, e.g. by killing it.

Once the port is free from the old binding, re-try starting the services.