Symptoms

Office 365 became unmanageable:

  1. All operations related to Office 365 application in CCPv1/UX1 and Online Store are hanging or failing with the message unknown error;
  2. All tasks related to Office 365 APS are failing with different errors;

  3. sitelog on Office 365 APS endpoint is filled with the error messages like the following:

    2019-05-06 08:51:43,092 <3836> [  93] ERROR aps_endpoint: AzureAuthorizationTokenHttp  server returned an web exception 'The remote server returned an error: (400) Bad Request.'.
System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at Parallels.Office365.Gateway.BaseHttpWebRequest.ExecuteAndHandleResponse[TResponse,TErrorResponse,TException](String Prefix, HttpWebRequest webRequest, String requestBodyForPrint, IJsonSettings jsonSettings, Func`4 errorAction) in c:\inetpub\wwwroot\O365AppPanama\App_Code\HttpWebRequest\BaseHttpWebRequest.cs:line 226
2019-05-06 08:51:43,092 <3836> [  93] ERROR aps_endpoint: Server returned body '{"error":"invalid_grant","error_description":"AADSTS700082: The refresh token has expired due to inactivity. The token was issued on 2019-02-03T15:22:19.6643819Z and was inactive for 90.00:00:00.\r\nTrace ID: 75718cf2-3f05-46f2-83b8-d96ca89c2e00\r\nCorrelation ID: 3617a41b-bc3a-47c8-a646-493dd1204692\r\nTimestamp: 2019-05-06 13:51:58Z","error_codes":[700082],"timestamp":"2019-05-06 13:51:58Z","trace_id":"75718cf2-3f05-46f2-83b8-d96ca89c2e00","correlation_id":"3617a41b-bc3a-47c8-a646-493dd1204692"}'.
  4. Operations to update refresh token for Office365 APS instance via PCP are hanging;

  5. console.log on the Management Node is filled with the following messages:

    190503 01:26:18,441 ERROR [io.undertow.request] (aps-service task-229) UT005023: Exception handling request to /aps/2/resources/555bd77d-dd7f-4224-ac58-32508c842fa0/getMSSubscriptionsInfoFast: com.parallels.pa.service.aps.api.APSEndpointExecutionException: AADSTS700082: The refresh token has expired due to inactivity. The token was issued on 2019-02-02T05:28:03.6345003Z and was inactive for 90.00:00:00.
...
190503 02:43:39,280 ERROR [io.undertow.request] (aps-service task-66) UT005023: Exception handling request to /aps/2/resources/72dc1c7e-c4ba-4fcd-ba1f-2140b4014630/checkAccount: com.parallels.pa.service.aps.api.APSEndpointExecutionException: Could not invoke endpoint url 'https://o365aeh.domain.tld:443/Office365gwAppcspmxpr/aps/office365app/72dc1c7e-c4ba-4fcd-ba1f-2140b4014630/checkAccount?accountId=1000003&planId=-1' of application instance with UUID '46223094-dea3-45b8-a1ff-d30995600fb0'. RESTEASY004655: Unable to invoke request

  6. In rare cases all operations on the OA platform became unmanageable along with presence of the big amount of the aforementioned errors in console.log and messages in core.log for lost transactions:

    May  3 09:49:30.903 : ERR [rest:542821 1:16536:7fadcf776700 SAAS ]: [ resolveCosControl] [APSC] The transaction with id '1540609877' is not found by unknown reason

New orders on activating or disabling Office 365 subscriptions may fail with the errors like:

Could not persist APS entity "http://parallels.com/aps/types/pa/subscription/1.0". Service Creation Failed: Platform error #error_code #-1, extype_id #-1, module_id #, Could not persist APS entity "http://parallels.com/aps/types/pa/subscription/1.0"

and

com.parallels.pa.service.aps.exception.APSInvocationException: Failed to initialize event accumulator

The errors above appear under the conditions of huge amount of simultaneous requests to Office 365 endpoint that are hanging, rendering OpenAPI and APS REST API interfaces unusable.

Cause

Expired token for Office 365 APS instances causes APS and Platform performance degradation due to software-related issue APSA-21162.

Resolution

Contact your Technical Account Manager or PTA team in order to trace the status of the issue APSA-21162.

To workaround the issue in case of only Office 365 unavailability:

  1. Cancel periodic tasks "Office 365 Synchronization with Office 365 Portal" for all instances of Office 365 application in PCP > Operations > Tasks > Periodic Tasks tab;

  2. Restart IIS service on the Office 365 APS endpoint server:

    iisreset
  3. Update refresh token for APS instances via PCP;
  4. Run previously canceled periodic tasks;

In case of whole platform unavailability with aforementioned symptoms it will be also required to restart OA services on the Management Node.

Internal content