Symptoms

Order Management components is not working, all request addressed to the component fail with error (for instance, GET/POST requests to "/aps/2/services/order-manager/orders").


From a log of Order Management kubernetes pod, it can be seen that OutOfMemory exception appeared after the following APS request:

01-09-2020;06:00:31,694 TRACE [uispan:90f07a88-89c1-456e-bb06-524d5e798c56 application-rest:task-58:604 a8n-order-management-com-ingrammicro-billing] c.i.i.l.components.trace.Tracer - TRACE FINISH APS CLIENT GET https://10.2.10.5:6308/aps/2/resources/e2262d05-2b8f-4c44-b2f4-c2900f9659c4/aps/links
java.lang.OutOfMemoryError: Java heap space




Cause


Software-related issue #OA-16606 "OutOfMemoryError in Order Management on loading high amount of APS links objects".


Resolution


The software-related issue will be fixed in future releases of Order Management application.


As a workaround, it is possible to temporary increase Java heap size for Order Management pod following steps below:


1. Check current heap size (in example below, minimum size is 512 MB and maximum size is 1024 MB):


[root@mn ~]# helm get values order-management | grep javaheapsize
javaheapsize: -Xms512m -Xmx1024m
[root@mn ~]# kubectl exec svc/order-management -- ps -C java -o args | grep -- -Xm
COMMAND
/usr/lib/jvm/java/bin/java -D[Standalone] -server -Xms512m -Xmx1024m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/jboss/wildfly/standalone/log -Xlog:gc*=debug:file=/opt/jboss/wildfly/standalone/log/gc.log:time,uptimemillis,tid:filesize=1M,filecount=5 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED --add-modules=java.se -Dorg.jboss.boot.log.file=/opt/jboss/wildfly/standalone/log/server.log -Dlogging.configuration=file:/opt/jboss/wildfly/standalone/configuration/logging.properties -jar /opt/jboss/wildfly/jboss-modules.jar -mp /opt/jboss/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/wildfly -Djboss.server.base.dir=/opt/jboss/wildfly/standalone -c standalone-full.xml -b 0.0.0.0

2. Check the current version of Order Management chart:

[root@mn ~]# helm get order-management | grep -m 1 chart
    chart: "ordermanagement-1.3.235"

3. Upgrade the release with increased values for Java heap:


helm upgrade order-management a8n/ordermanagement --version 1.3.235 --reuse-values --set javaheapsize="-Xms512m -Xmx2048m" --recreate-pods --wait

4. Check that new values were applied by executing commands from step one:


helm get values order-management | grep javaheapsize
kubectl exec svc/order-management -- ps -C java -o args | grep -- -Xm