March 2021

Exchange hybrid is the only Microsoft migration method that allows you to move mailboxes in both directions between Exchange and Office 365.

I'm working with a customer who is offboarding their mailboxes from Exchange Online back to Exchange Server 2016. Several mailboxes cannot be migrated back to on-premises due to the following error:

You cannot migrate mailbox '2f15ec3c-b024-4a25-2fdf-bfda9fd806ba' off of Office365 while the mailbox has a connected account enabled. To successfully remove the connected accounts, run the following command: 'Get-Subscription -Mailbox MailboxID -Aggregati

Note the truncated command, which isn't very useful.

The Get-Subscription cmdlet let's you view the properties of an existing subscription configured in a user's cloud-based mailbox. This cmdlet is used by Outlook on the web Options to display the list of email subscriptions that the end user has, such as POP, IMAP, Facebook, and LinkedIn.

The exact issue with the mailbox can be determined by running the following in Exchange Online PowerShell:

Get-Subscription -Mailbox user -AggregationType all | select DisplayName, Name, AggregationType, UserId, LastModifiedTime, LastSuccessfulSync, CreationType, Status, StatusDescription, DetailedStatus

We can see that the user has configured a subscription to LinkedIn so that Outlook can do a PeopleConnection lookup. This service was closed down by Microsoft over 3 years ago, but the subscription on the mailbox remains.

Run the following cmdlet in EXOPS to remove the subscription from the mailbox:

Get-Subscription -Mailbox UserId -AggregationType All | Remove-Subscription

Now you can resume the mailbox migration to complete the offboard.

Important Things to Know

The vulnerability affects ALL versions and update levels of Exchange.
The attack starts with an unauthenticated request to an Exchange server, so if your Exchange Server is not publicly accessible, less risk can be assumed. You're still open to internal attacks, though. PATCH YOUR SERVERS.
Security updates have been released for the following Exchange versions and Cumulative Update levels:

Exchange 2010 SP3
Exchange 2013 CU23
Exchange 2016 CU18
Exchange 2016 CU19
Exchange 2019 CU7
Exchange 2019 CU8

If your Exchange Servers are not at these CU levels, you need to get on the latest CU for your version before you can install the security update.

Be aware the .NET Framework and Visual Studio C++ runtime prerequisites may be required to update your server to the latest CU. The Exchange CU setup will block installation until the prerequisites are satisfied.
.NET Framework installations and updates can peg your CPU for a while after restarting. Plan accordingly.

You MUST run the security update from an ELEVATED CMD prompt. Admins who ignore this have reported that it breaks ECP.
Hybrid customers are affected, as well. Apply these updates to your hybrid management servers.

Best Practices

Apply the patches ASAP.
Always restart the Exchange server BEFORE you install the security updates or upgrade the CU. That way you know that the server reboots properly before you install the updates. It also releases any locked files.
The update normally takes 15-20 minutes to install. Always restart the Exchange Server AFTER installing the security updates or upgrading the CU (whether it prompts you to, or not). I find that it doesn't warn you to restart about 50% of the time on the dozen or so servers I've upgraded.
Do not "batch" updates. For example, don't upgrade Exchange 2016 CU17 to CU19 and immediately install the security patch without rebooting. Restart Exchange Server after every update.
Always keep your Exchange Server up to date with the latest (or second to latest) CU so you can more easily install urgent updates like these.

Helpful Tips

Run the following from an elevated PowerShell or EMS console so you can right-click an MSI or MSP file and "Run as Administrator":

New-ItemProperty registry::HKEY_CLASSES_ROOT\Msi.Patch\shell\runas\command -Name "(Default)" -Value '"%SystemRoot%\System32\msiexec.exe" /p "%1" %*' -PropertyType ExpandString -Force
New-ItemProperty registry::HKEY_CLASSES_ROOT\Msi.Package\shell\runas\command -Name "(Default)" -Value '"%SystemRoot%\System32\msiexec.exe" /p "%1" %*' -PropertyType ExpandString -Force

Use the following PowerShell script to check for compromise. Run elevated. This script does not work on Exchange 2010.

Start-Transcript -Path "Check-Compromise-$env:COMPUTERNAME.log" -Force
Write-Host "Checking for CVE-2021-26858 exploitation..."
findstr /snip /c:"Download failed and temporary file" "%PROGRAMFILES%\Microsoft\Exchange Server\V15\Logging\OABGeneratorLog\*.log"
Write-Host "Checking for CVE-2021-26857 exploitation..."
Get-EventLog -LogName Application -Source "MSExchange Unified Messaging" -EntryType Error | Where-Object { $_.Message -like "*System.InvalidCastException*" }
Write-Host "Checking for CVE-2021-27065 exploitation..."
Select-String -Path "$env:PROGRAMFILES\Microsoft\Exchange Server\V15\Logging\ECP\Server\*.log" -Pattern 'Set-.+VirtualDirectory'
Write-Host "Checking CVE-2021-26855 exploitation..."
$logs = (Get-ChildItem -Recurse -Path "$env:PROGRAMFILES\Microsoft\Exchange Server\V15\Logging\HttpProxy" -Filter '*.log' -ErrorAction SilentlyContinue).FullName | sort -Descending
$logCount = $logs.Count
foreach ($log in $logs) {
$i++
$percentage = $i / $logCount * 100
$perc = $percentage.ToString("#")
Write-Progress "Search in progress" -Status "$perc% Complete:" -PercentComplete $percentage
Import-Csv -Path $log -ErrorAction SilentlyContinue | Where-Object { $_.AuthenticatedUser -eq "" -and $_.AnchorMailbox -like 'ServerInfo~*/*' } | select DateTime, AnchorMailbox
}
Stop-Transcript

Review the Check-Compromise-<computername>.log file for results. You can ignore any errors that say something like, "The member '40' is already present." See HAFNIUM targeting Exchange Servers with 0-day exploits for more details.

If your update fails for some reason, you may find that all the Exchange services are stopped and disabled. If you try to rerun the update it will fail again for this reason. Run the following to return the Exchange service to automatic, then reboot and try again ELEVATED.

Get-Service -Name MSExchangeDelivery, MSExchangeRepl,MSExchangeRPC,MSExchangeFastSearch, MSExchangeThrottling, wsbexchange,MSExchangeSubmission, MSExchangeMailboxReplication, MSExchangeMailboxAssistants, MSExchangeIMAP4BE, MSExchangeImap4, MSExchangeIS, MSExchangeDagMgmt, MSExchangeDiagnostics, MSExchangeFrontEndTransport, MSExchangeADTopology, MSExchangeAntispamUpdate, MSExchangeUM, MSExchangeEdgeSync, MSExchangeHM, MSExchangeHMRecovery, MSExchangeServiceHost, MSExchangeTransport, MSExchangeTransportLogSearch, FMS, HostControllerService, winmgmt, remoteregistry, w3svc, iisadmin | Set-Service -StartupType automatic

Whenever a new version of .NET Framework is installed or a .NET Framework update is applied, the server CPU will peg at 90-100% utilization after reboot for up to 40 minutes while it recompiles MOFs. This process is also called NGEN. You will see mscorsvw.exe processes chewing up CPU. The same happens after you install an Exchange update, since Exchange Server is written in .NET.

Use the 7318.DrainNGenQueue.wsf script to speed up .NET Framework recompiling performance by allowing it to use multiple threads and up to 6 cores. Run it after any .NET Framework installation or update.

Pages

You cannot migrate mailbox off of Office365 while the mailbox has a connected account enabled

Please join me for "HAFNIUM Exchange Server Hack: Why Patching Isn't Enough & Where to Start Hunting" webinar

New Article: How to Decide Between Azure AD Connect and Azure AD Connect Cloud Sync

URGENT: Patch your Exchange Servers NOW!

Important Things to Know

Best Practices

Helpful Tips