Symptoms

When upgrading your CloudBlue Commerce installation from version 20.5 to version 21.x, oss-init-job Pods fail with the "Wrong number of applications to upgrade" error. For example:

# kubectl -n cbc get pods | grep oss
oss-init-job-2gmtw                                              0/1     Error     0          8m41s
...
oss-init-job-xm7dv                                              0/1     Error     0          14m
oss-node-0                                                      1/1     Running   0          17m

# kubectl -n cbc logs oss-init-job-xm7dv
...
2022-07-23 13:12:07.307 [INFO] Installing system APS packages ...
...
2022-07-23 13:12:07.491 [DEBUG] SQL: SELECT ap.version, ap.release FROM aps_package ap JOIN aps_application aa ON aa.package_id = ap.id WHERE ap.name='User and Account Management' (())
2022-07-23 13:12:07.492 [INFO] Upgrading system APS package: "User and Account Management" from 20.4.0-144 to 21.6.0-144
2022-07-23 13:12:07.492 [DEBUG] call method pem.APS.upgradeApplication({'package_url': 'file:///build/packages/provide/uam.application-21.6.0-144.app.zip'})
2022-07-23 13:12:21.190 [DEBUG] return {'status': -1, 'error_message': 'Wrong number of applications to upgrade', 'extype_id': -1, 'module_id': '', 'error_code': -1, 'properties': {'package_url': 'file:///build/packages/provide/uam.application-21.6.0-144.app.zip'}}
2022-07-23 13:12:21.190 [DEBUG] Traceback saved:
(<class 'poaupdater.openapi.OpenAPIError'>, OpenAPIError('Wrong number of applications to upgrade',), <traceback object at 0x7f37f0aaf550>)
Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "post_install.py", line 38, in main
    install_system_packages()
  File "post_install.py", line 38, in main
  File "post_install.py", line 29, in install_system_packages
    install_system_packages()
    post_upgrade.upgrade_system_packages()
  File "post_install.py", line 29, in install_system_packages
  File "post_upgrade.py", line 139, in upgrade_system_packages
    _upgrade_package(pkg, oss_pkgs[pkg])
  File "post_upgrade.py", line 56, in _upgrade_package
    post_upgrade.upgrade_system_packages()
    open_api.pem.APS.upgradeApplication(package_url='file://' + pkg)
  File "post_upgrade.py", line 139, in upgrade_system_packages
  File "/usr/lib/python2.7/site-packages/poaupdater/openapi.py", line 239, in __call__
    raise OpenAPIError(resp)
OpenAPIError: Wrong number of applications to upgrade

    _upgrade_package(pkg, oss_pkgs[pkg])
  File "post_upgrade.py", line 56, in _upgrade_package
    open_api.pem.APS.upgradeApplication(package_url='file://' + pkg)
  File "/usr/lib/python2.7/site-packages/poaupdater/openapi.py", line 239, in __call__
    raise OpenAPIError(resp)
poaupdater.openapi.OpenAPIError: Wrong number of applications to upgrade

Cause

There are two application instances of the same system APS application, which causes the upgrade script in those Pods to fail. For example:

oss=# SELECT * FROM aps_application_instances WHERE app_id IN (SELECT app_id FROM aps_applications WHERE name='UAM');

 app_instance_id | app_id |  app_ver   | real_app_ver | app_conf_id |                 uri                 | prov_type | master_app_instance_id | owner_id | resource_id | status |                 uuid
-----------------+--------+------------+--------------+-------------+-------------------------------------+-----------+------------------------+----------+-------------+--------+--------------------------------------
           10237 |    474 | 20.4.0-144 |              |             | [https://127.0.0.1:8081/pa/rest/fake] | e         |                        |        1 |             | n      | 599c5af2-0029-40aa-aa59-e15124b5511b
            3633 |    474 | 20.5.0-144 |              |         261 | [https://127.0.0.1:8081/pa/rest/fake] | e         |                        |        1 |             | n      | e7258b20-0109-4f2d-96f5-fc4b117dac3a
(2 rows)

In this example, UAM is the name of the system APS application that has two application instances. That name was obtaned based on the log records above:

2022-07-23 13:12:07.492 [DEBUG] call method pem.APS.upgradeApplication({'package_url': 'file:///build/packages/provide/uam.application-21.6.0-144.app.zip'})
2022-07-23 13:12:21.190 [DEBUG] return {'status': -1, 'error_message': 'Wrong number of applications to upgrade', 'extype_id': -1, 'module_id': '', 'error_code': -1, 'properties': {'package_url': 'file:///build/packages/provide/uam.application-21.6.0-144.app.zip'}}

Resolution

1. Remove the application instance with the least version:
   1. Enter the oss-node-0 Pod:

      kubectl -n CBC_NAMESPACE exec -ti oss-node-0 bash

   2. In that Pod, run the following Python script:

      from poaupdater import openapi
      api = openapi.OpenAPI()
      api.pem.APS.unproviderApplicationInstance(application_instance_id=APP_INSTANCE_ID_OF_APP_INSTANCE_WITH_LEAST_VERSION)

2. Uninstall the oss component:

   helm -n CBC_NAMESPACE delete oss

3. Run helmfile-deploy again:

   helm -n CBC_NAMESPACE upgrade helmfile-deploy a8n/helmfile-deploy --set configYaml=$(cat config.yaml | base64 -w0) --set namespace=CBC_NAMESPACE --set dockerrepo=odindevops-a8n-docker-21-public.jfrog.io