## Symptoms


Periodic task "Synchronize resource usage for APS application instance #12 (Azure Cloud Solution Provider)" fails with the following error message:


Could not get resource usage for resource with UUID '0647d955-818a-4fc0-a18c-302de28c4f57': '500 Internal Server Error' received from 'GET https://192.168.0.150:10443/azure/aps/tenants/0647d955-818a-4fc0-a18c-302de28c4f57': Unexpected empty response for subscription utilization request.
500 Internal Server Error


Or:


Body: com.parallels.pa.service.aps.api.APSEndpointExecutionException: '500 Internal Server Error' received from 'GET https://192.168.0.150:10443/azure/aps/tenants/be0f6eeb-7cf1-4194-a833-f83120ce45b0': Unexpected empty response for subscription utilization request.
Could not get resource usage for resource with UUID '88cfe31b-2ad2-42fd-adf7-61287ed8202e': '501 Not Implemented' received from 'GET https://192.168.0.150:10443/azure/aps/tenants/88cfe31b-2ad2-42fd-adf7-61287ed8202e': The enumerator does not have a current value
501 Not Implemented

It can be seen in azure.log on Azure CSP endpoint that failure appears after "204 No Content" response is received on utilization request:


2020-03-10 15:27:05,654 DEBUG [com.odin.azure.api.PartnerCenterApi] (azure task-72) <PCSDK> The request: 'GET https://api.partnercenter.microsoft.com/v1/customers/21c9***ef1a9a551/subscriptions/CC9***77A6/utilizations/azure?seek_operation=Next' has been completed: 'HTTP/1.1 204 No Content'
- headers:          
        Content-Length: 0       
        Retry-After: Tue, 10 Mar 2020 15:42:05 GMT      
        MS-CorrelationId:       
        MS-RequestId:           
        X-Locale: en-US         
        MS-CV: QNvg/9kNv0ua3gzg.0           
        MS-ServerId: B10000OQ   
        Request-Context:       
        Date: Tue, 10 Mar 2020 15:27:05 GMT 
2020-03-10 15:27:05,654 ERROR [com.odin.azure.server.ErrorsHandler] (azure task-72) <APS-APP> Error: java.lang.UnsupportedOperationException: The enumerator does not have a current value



## Cause


Response "204 No Content" is not expected to be received by the Azure CSP application endpoint. Such response is given by Microsoft when usage utilization data is not prepared yet.


## Resolution


In order to avoid similar failures, it is possible to set a delay in usage collection on Azure CSP endpoint:


1. Modify /usr/local/azure/site/config/config.json configuration file and set following values for corresponding parameters:


Important! Changes should be applied in accordance with current version in Azure CSP application installed in the platform.


For Azure CSP 7.7 and below:

    "usageLatencyHours": {
        "max": 48,
        "step": 2,
        "min": 2
    },


For Azure CSP 7.7.1:


    "usageLatencyHours": {
        "max": 3,
        "step": 1,
        "min": 3
    },


For Azure CSP 7.7.2 and above:

    "usageLatencyHours": {
        "max": 48,
        "step": 2,
        "min": 2
    },




2. Restart Azure application endpoint:


service azure-app-server restart


Definitions of values provided above:


"min" - minimal difference between current time and time set in utilization request.

"max" - maximal difference between current time and timestamp set in utilization request.

"step" - in case if usage is not ready yet, the different between current time and timestamp in utilization request is increased by amount of hours specified for this parameter.


Example of how it works:


For instance, current date/time is 2020-03-10 13:00:00. 


"min" is specified as 2, which means that difference between current time and time for utilization request is two hours:


https://api.partnercenter.microsoft.com/v1/customers/31c0***1a8a645/subscriptions/CC9F***377A6/utilizations/azure?start_time=2020-03-10T00:00:00Z&end_time=2020-03-10T11:00:00Z&granularity=Hourly&show_details=true&size=500'


Above you can see that "end_time" is specified with two hour difference from the current time. In case if usage is not prepared yet by Microsoft and error is given on response, difference is increased by two more hours because "step" value is specified as "2":

 

https://api.partnercenter.microsoft.com/v1/customers/31c0***1a8a645/subscriptions/CC9F***377A6/utilizations/azure?start_time=2020-03-10T00:00:00Z&end_time=2020-03-10T09:00:00Z&granularity=Hourly&show_details=true&size=500'

In case if this request also fails, difference is increased by two more hours. That happens up until maximum allowed difference set as 48 hours is reached.