Fix for MSExchange Unified Messaging Event ID 1423 "A TLS API failure occurred. Error = 0x80090331"

I was working with a customer who is implementing Unified Messaging in an Exchange 2013 CU7 environment with Lync 2013. All Exchange servers are running on Windows Server 2012 R2. Exchange UM was configured properly, including setting the Unified Messaging Call Router and Unified Messaging services to "Dual" and configuring valid SSL certs for both UM services. The dial plans and Auto Attendants were created and they started to test.

Calls made to the UM enabled user would go to voicemail, the greeting would play, and the caller could record a voicemail, but the voicemail would not be delivered to the UM user's mailbox. We turned up UM logging using the following cmdlet:

Get-EventLogLevel "msexchange unified*" | Set-EventLogLevel -Level Expert

Another voicemail test showed the following MSExchange Unified Messaging event ID 1423 - A TLS API failure occurred. Error = 0x80090331.



A quick look in the %ExchangeInstallPath%\UnifiedMessaging\voicemail folder on the mailbox server hosting the UM user's mailbox showed all the voicemail WAV files queued for delivery.

I'm familiar with error code 0x80090301 on this event ID, which is caused by too many root certificates in the Trusted Root Certification Authorities store, but this code and error sting is different. 0x8009331 means, "SEC_E_ALGORITHM_MISMATCH - The client and server cannot communicate, because they do not possess a common algorithm." Definitely sounds like a TLS negotiation problem.

After double-checking the UM configuration and the SSL certificate configuration used by the UM services it was found that TLS 1.0 was disabled for clients on the Exchange 2013 servers. The registry key HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client\DisabledByDefault was set to 1. This key doesn't exist by default and was not being configured via GPO, so it must have been configured in their server build. Setting the value to 0 (or deleting the DisabledByDefault key) fixed the problem.

The problem occurred because the UM server acts as both a client and a server to itself. The UM service acts as a client when it reads the voicemail WAV files from the voicemail folder and uses TLS to do so.

Note that there were several improvements to TLS and SSL in Exchange 2013 CU8 and Exchange 2010 SP3 RU9. These versions support TLS 1.1 and 1.2, which should also work. Please read Scott Landry's excellent article, Exchange TLS & SSL Best Practices, for more information.

Read more ...

Exchange 2013 Migration Batch Completes Successfully, But Mailbox is Not Moved

Here's another weird one. I'm testing a mailbox move from Exchange 2013 to Exchange 2016 (beta) for a mailbox with an in-place archive. I can create a new migration batch for the user and it completes in short order, but the mailbox and archive are not moved. The number of Total, Synced, Failed, and Successful mailboxes for the batch is zero. I get a notification email saying the batch completed successfully, but no mailboxes were moved or synced.

"Successful" Migration Batch Notification

When I look at the details of the migration batch in EMS I see that the ValidationWarningCount is 1 and the ValidationWarnings show, "Error:The user already exists, but the migration batch that includes it couldn't be found. Before you try migrating the user within a batch again, please remove the existing user by running the Remove-MigrationUser cmdlet."

Migration Batch Validation Warnings

Whoever wrote that error message deserves a raise. MUCH better than the "Error code 2" I'm used to getting.

A "MigrationUser" object is created for each user who is in the process of being migrated or the migration has not been completed, so it remains in a syncing state. I ran the Get-MigrationUser cmdlet which revealed that the mailbox was indeed "stuck" in a migration since 7/20/2013(!!) with no other issues.

This mailbox has been "migrating" for over two years!

So, I removed the migration user using the Remove-MigrationUser cmdlet. I needed to use the -Force parameter since the move request no longer exists.

Remove-MigrationUser clears the way to re-run the migration

This fixed the problem and I was able to move the mailbox and archive successfully. This condition would prevent a mailbox from moving to any other database or Office 365. It's strange that Exchange doesn't detect this when the batch request is created.

Read more ...

Either way, it's OWA

<rant>

Way to kill off a well-known brand. According to this article on the Office Blog, Microsoft marketing has decided to rename the well known Outlook Web App (OWA) to Outlook on the web (Ootw). Note the small "w" in web - don't want to be accused of making the web a Microsoft brand. Ootw just rolls off the tongue, doesn’t it? Apparently Microsoft marketing gets paid by the consonant.

According to Scott Davis at Microsoft:

"The goal here is to simplify the branding and providing consistency across platforms. The brand is really just ‘Outlook’ – ‘on the web’ is simply a descriptor to help users understand which Outlook we are referring to. For example, we also have Outlook on iOS, Android and soon, Windows 10 Mobile. If you look in the App Store/Google Play, these apps are simply branded “Outlook”. In our marketing or help content, we may say Outlook on iOS, Outlook on Android, Outlook on the web, etc. But in each case, the brand is just Outlook. Most importantly, users in all cases will just see the Outlook brand when they are using the products."

With Outlook now being called just “Outlook” on Windows, iOS, Android, Windows modern app for Windows 10 (coming soon), look forward to many fun-filled support calls that go like this: “Are you running Outlook, Outlook, or Outlook?” “No, I’m running OUTLOOK.”

Utter #FAIL

</rant>

Read more ...

Fix for Server Manager Error: "Online - Cannot get performance counter data"

One of the interesting things about having a home lab is you get to break things in ways that no one thought possible. I'll give a nickle to the next person who has this happen to them.

I have three Exchange servers running Windows Server 2012 R2. Performance counters have been started on all three servers (see https://technet.microsoft.com/en-us/library/hh831394.aspx). One server had two IP addresses configured for one NIC so I could do some testing. When I removed the second IP address from the single NIC and restarted the server Server Manager complained that it could not refresh that server.

When I clicked Manageability in Server Manager it showed "Online - Cannot get performance counter data." Re-adding the secondary IP and restarting the server doesn't help.

Lots of troubleshooting later (involving Perfmon, Bing-fu, and swearing) I discovered the following fix:

1. Open an elevated CMD prompt to C:\Windows
2. Run lodctr /R to rebuild the perf registry strings and info from scratch based on the current registry settings and backup INI files. If this works, you're done. In my case, it resulted with the error, "Error: Unable to rebuild performance counter setting from system backup store, error code is 2." Very helpful. :-|
3. Change to the C:\Windows\SysWOW64 folder and run lodctr /R again. This time I got "Info: Successfully rebuilt performance counter setting from system backup store"
4. Run winmgmt /resyncperf to register the system performance libraries with WMI and then refresh Server Manager to see that the problem is resolved.

One of the reasons I run this blog is to maintain a memory of esoteric things like this. I doubt I'll ever see it again.

Read more ...