ISSUE: Remad.Exchange Service hangs in 'Starting' status indefinitely

Last modified: 2 years ago

Overview

Consider the following scenario. Exchange ingestions have halted and the Exchange module is offline when looking in the portal. When logging onto the Bridgehead server, it is unresponsive and requires a reboot. Upon reviewing the event logs, there is an error related to the Exchange service exhausting all system memory and crashing. After a reboot, the service then hangs in a 'starting' state indefinitely.

Reviewing the backend logs and/or logs on the Bridgehead server, it is flooded with the error below repeatedly:

Error in the Exchange backend log:

EXCEPTION: [500|https://tools.ietf.org/html/rfc7231#section-6.6.1|An error occurred while processing your request.|Unable to find mapping with ID 2579||00-ebcf19af7e0221f78b030d02c84191ec-c88751baf3a88e7b-00|]

Error in the Evcomplete.Exchange log on the Bridgehead:

Cause

While the root cause is unknown, the interim explanation is the Exchange service is expecting a result for this mapping id and does not receive one and keeps retrying every 5 seconds. Each retry causes an entry in the TempCache folder locally on the server.

Solution

Note: this is a temporary solution to allow the service to start so ingestions can continue. Root Cause needs to be determined.

1. On the Bridgehead server, browse to: D:\program files\cloudficient\tempcache\exchange

2. Make a copy of the Exchange folder

3. End Task for the Exchange service in the Task Manager

4. Search the Exchange folder for the string highlighted in Red in the screenshot above. It should find thousands of them.

5. Once the search completes, delete all these files.

6. Now start the Remad.Exchange service and confirm it starts successfully and it's showing online in the portal