Symptoms
Cross-forest Exchange migration is being performed for a client (for instance, migration of organization from Exchange 2010 to Exchange 2016 servers).
Task "Migration: Prepare for Exchange organization #*** upgrade" fails with the following error:
APS resource for '/aps/2/application/mail-contact/***' with id '***' was NOT deleted. Error code: 403.
Or with the following error:
APS resource for '/aps/2/application/mail-mailbox/22f14f9c-4bcc-432c-bde5-4f069d6886ca' with id '3385998' was NOT deleted. Error code: 403.
It could be seen in core.log that task fails due to APS token expiration:
Aug 28 16:18:26.062 : ERR [task:247177118:164643 1:4906:7f899b7de700 SAAS 1570024452]: [ APSC] REST Error: DELETE /aps/2/application/mail-contact/798cddc4-1e31-483d-b5d9-c00ebbf951ce code=403: Token expired
Aug 28 16:18:26.062 : DBG [openapi:2175308 p:-default-threadpool;-w:-Idle:586 pau]: c.p.p.tracer entry: com.parallels.pa.service.branding.ejb.BrandedServiceBean.findBrandedServices(com.parallels.pa.ser
vice.branding.api.BrandableServiceLookupFilter@1ea25b33) correlationId openapi:2175308
Aug 28 16:18:26.062 : DBG [task:247177118:164643 1:4906:7f899b7de700 SAAS 1570024452]: [ HTTP] IN Response headers: 'Date:Fri, 28 Aug 2020 15:18:26 GMT' 'Connection:Close' 'Content-Type:application/
json' 'Content-Length:69'
Aug 28 16:18:26.062 : DBG [openapi:2175308 p:-default-threadpool;-w:-Idle:586 pau]: c.p.p.s.b.e.BrandedServiceBean findBrandedServices for brand 0 account 1049155 service ExchProtocols
Aug 28 16:18:26.062 : INF [task:247177118:164643 1:4906:7f899b7de700 SAAS 1570024452]: [ HTTP] IN 0.0.0.0 [69] "DELETE /aps/2/application/mail-contact/798cddc4-1e31-483d-b5d9-c00ebbf951ce HTTP/1.0"
[0] "403 Forbidden" 26ms
Aug 28 16:18:26.062 : TRC [task:247177118:164643 1:4906:7f899b7de700 lib 1570024452]: [ CosTransactions::Current_impl::suspend] ===> ENTRY
Aug 28 16:18:26.062 : DBG [openapi:2175308 p:-default-threadpool;-w:-Idle:586 pau]: c.p.p.tracer entry: com.parallels.pa.service.branding.ejb.BrandingManagerBean.findAccountBrand(1049155) correlationId openapi:2175308
Aug 28 16:18:26.062 : DBG [task:254955649:165113 p:-default-threadpool;-w:-Idle:588 pau]: c.p.p.s.d.e.DBServiceBean ' SELECT mailbox_id, own_act_size FROM exch_mailboxes_du WHERE mailbox_id IN (? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ? ) AND remove_in_process = ?'. Parameters: [3707269, 3707278, 3707280, 3707292, 3707242, 3707276, 3707283, 3707344, 3707
333, 3707334, 3782113, 3782156, 3782108, 3675482, 3717804, 3717803, n] [sqlpid: 47509]
Aug 28 16:18:26.063 : INF [task:247177118:164643 1:4906:7f899b7de700 lib ]: [ CosTransactions::Current_impl::suspend] Transaction 1570024452 suspended
Aug 28 16:18:26.063 : TRC [task:247177118:164643 1:4906:7f899b7de700 lib ]: [ CosTransactions::Current_impl::suspend] <=== EXIT [0.000428]
...
Aug 28 16:18:26.068 : ERR [task:247177118:164643 1:4505:7efde9ffb700 lib 1570024452]: [ SDK::RemoveMailRecipientsObserver::onRemoveMailRecipients] ExSystem: module_id:'SaaS', ex_type_id:'37',Message:'APS resource for '/aps/2/application/mail-contact/798cddc4-1e31-483d-b5d9-c00ebbf951ce' with id '3078195' was NOT deleted. Error code: 403.', deprecated_codes = (0, 0), properties = { resURI: '/aps/2/application/mail-contact/798cddc4-1e31-483d-b5d9-c00ebbf951ce',object_id: '3078195',error_code: '403', }
Aug 28 16:18:26.068 : ERR [task:247177118:164643 1:4505:7efde9ffb700 lib 1570024452]: [ SDK::RemoveMailRecipientsObserver::onRemoveMailRecipients] failed to notify observer sc_id 140 [SDK::RemoveMailRe
cipientsObserver] (CORBA::Exception)Cause
Software-related issue #OA-16455 "Cross-forest upgrade fails for Exchange organization with ~10k contacts"
Resolution
The issue is considered to be fixed in future releases of CloudBlue Commerce. Contact CloudBlue Technical Support in order to apply a workaround for the issue.