Warning - High number of corrupt items in hybrid migrations

Wednesday, February 14, 2018
UPDATE: This issue has been resolved, but the outcome was less than optimal. If you configured your mailboxes to migrate with a high message corruption limit and the mailbox move succeeded, those mailboxes will have data loss. Microsoft's "remediation plan" is the following:
What action do I need to take? If your migration failed, we recommend that you cancel and re-submit any affected migration requests and proceed with any migrations that were postponed. For migrations that completed with missing items, we have identified a remediation plan that admins can apply that will assist with restoring missing calendar items.

Admins will have a three-step process that they will need to take:
  1. Offboard the mailbox. 
  2. Restore calendar content into the mailbox using a New-MailboxRestoreRequest command. 
  3. Onboard the restored on-premises mailbox back into Exchange Online. 

My original post follows:

I'm publishing this article as a public service announcement since Microsoft is not warning customers in the Office 365 Service Health Dashboard.

Beginning last week, we noticed that hybrid mailbox moves to Exchange Online started failing with a very high number of corrupt items, usually calendar items. The default limit for corrupt items is 10 in a hybrid mailbox migration. It's not unusual for some mailboxes have one or two corrupt mail items, usually calendar items, for mailboxes that have been around for a long time. But suddenly mailbox moves are starting to fail with hundreds of corrupt items.

Migration Batch Status

Migration Report Details 
I've seen this myself in my own tenant, as well as other customer's tenants. It's also being reported by customers in the Microsoft Online: Exchange Online forum. We know it's not a problem with the mailboxes because we can move them between Exchange on-prem databases without any corrupted items being detected.

I and other MVPs have opened support tickets, but they seem to have fallen into black holes. We're getting very little response and the issue still doesn't show in the O365 SHD as of the time of this article. So far, we've found that tenant versions 15.20.485.17, 15.20.506.17, and 15.20.527.17 are affected. To find out which version your tenant is, run the Get-OrganizationConfig | ft AdminDisplayVersion cmdlet in remote PowerShell.

Determine Tenant Version
WARNING: A lot of customers simply increase the corrupt item limit high enough to get the mailbox move to succeed, but that may mean a lot of data loss. I do not recommend that you do this until this issue is fixed.

If you have increased the corrupt item limit to get the migrations to complete, you can retrieve the non-migrated data from the soft deleted mailbox on-premises. See Connect or restore a deleted mailbox for details.

Microsoft has finally updated the Service Health Dashboard to show there's a problem. And they indicate that this affects all mailbox migrations since January 30, 2018!

It took over two full weeks for this issue to service on the SHD. I don't know about you, but I think this is totally unacceptable.

No comments:

Post a Comment

Thank you for your comment! It is my hope that you find the information here useful. Let others know if this post helped you out, or if you have a comment or further information.