Big Exchange News!

Lots of exciting announcements are being made today about Exchange Server on the EHLO Blog.

• Free Exchange 2019 hybrid keys are being made available. This, along with the public availability of Exchange Server 2019 bits, means that customers can deploy Exchange 2019 hybrid management servers for free!

• A new Exchange Management Tools update is being announced. This will allow customers who have completed their migration to Microsoft 365 Exchange Online an option to turn off (not remove!) their last Exchange Server. There are several large caveats to this -- The solution is PowerShell only, it does not support RBAC, and there is no auditing available. I'm working on a blog article that does a walkthrough and explains all the details.

• The Hybrid Configuration Wizard is making several improvements, including MFA support.

• Microsoft is changing the updates delivery model for Exchange Server to bi-annually, rather than (roughly) every quarter. This will allow customers more time for testing and deployment between releases.

• Exchange Server 2019 is adding Windows Server 2022 support. Exchange 2019 CU12 and above can be installed on Windows Server 2019 or Windows Server 2022.

• With the latest CUs, Exchange Server 2013/2016/2019 now supports Windows Server 2022 Active Directory environments. Exchange customers no longer need to put off upgrading their Domain Controllers.

• New Microsoft Bounty Program for Exchange Server. A security vulnerability bounty program for Microsoft Exchange Server is being launched to help keep Exchange Server secure for all customers.

Read all about these exciting changes here: Released: 2022 H1 Cumulative Updates for Exchange Server

Read more ...

Yes, Virginia, You Can Buy Exchange 2019!

In the Beginning

When Exchange 2019 was announced at Microsoft Ignite 2019, it was also announced that Exchange Server 2019 and its Cumulative Updates would be available only through the Volume Licensing Service Center (VLSC). It was explained that large enterprise customers were asking for security, reliability, and dependability. They want all the things that mean Exchange runs as a mission critical application.

Welcome to Exchange Server 2019! - BRK2176

As Greg Taylor, then Director of Product Marketing for Exchange Server/Online, said at the time, "For those customers who still want to stay on premises, that's the reason we built Exchange 2019. And that's also the reason why we are only going to distribute Exchange 2019 to those customers through Volume Licensing."

As anticipated, Exchange Server 2019 RTM and CUs 1-8 were only available through the VLSC and to developers for testing and application development through MSDN. And for the first time ever, the current version of Exchange Server was no longer available on the Office Servers Evaluation Center.

To access Exchange Server 2019 through the VLSC, customers must have an active agreement in one of the following Microsoft Volume Licensing programs:

• Microsoft Enterprise Agreement (500+ seats)
• Microsoft Products and Services Agreement (250+ seats)
• Microsoft Open Value Agreement (5-499 seats)

See the Compare Microsoft Volume Licensing Programs resource document for full details.

I've found that most small-midsize customers mistakenly think that access to the VLSC requires an Enterprise Agreement or large minimum spend requirement. Plus, most of these customers buy licenses through a third-party license provider, like a Cloud Solution Provider or licensing distribution partners. Few of the customers I speak with have actually entered into a licensing agreement directly with Microsoft.

The VLSC requirement imposed a barrier that prevented these customers from accessing Exchange Server 2019. And if non-VLSC customers cannot get Exchange 2019, it means that Exchange 2016 is the latest version they could use for hybrid management after they have completed their migration to the cloud.

As we know, an Exchange server is still required for Exchange recipient management even after all mailboxes have been moved to Microsoft 365 since Active Directory is still the Source of Authority for hybrid customers.

Then HAFNIUM happened...

In March 2021 a state-sponsored hacking group called HAFNIUM targeted Exchange Servers around the world by exploiting zero-day vulnerabilities. Threat actors gained access to email servers and installed malware to facilitate long-term access to victim environments and to perform data exfiltration.

Microsoft quickly responded to HAFNIUM by releasing Security Updates (SUs) that patched the Exchange Server vulnerabilities and a short time later included these fixes in the March 2021 Quarterly Exchange Updates. In an effort to ensure that all customers could get and stay up-to-date, it was decided to publish Exchange Server 2019 CU9 and future CUs to the Microsoft Download Center in addition to the VLSC.

What This Means to Hybrid Customers

Exchange hybrid customers who have completed their migration to the cloud can now use Exchange Server 2019 as their hybrid management server. All customers can now run the latest version of Exchange server with the most recent CUs and SUs by downloading them from the Download Center, even with a Volume Licensing agreement.

CU's are build-to-build upgrades and contain a full server installation, so the latest CU can be used for a fresh installation. Always check the Exchange Team Blog for details on the latest CU. All customers, including hybrid customers, should keep their Exchange servers up to date using the N-1 support statement (the current and previous CUs and SUs are supported).

Keep in mind that currently there is no free Exchange hybrid license available for Exchange 2019 like there is for Exchange 2013/2016, so customers will need to license their Exchange Server 2019. See Big Exchange Announcements!

Customers with Exchange Server 2010 must keep in mind that Exchange Server 2019 will not install if Exchange 2010 is in the environment. Those customers must transition to Exchange Server 2016 and decommission Exchange 2010 before installing Exchange 2019.

Read more ...

Windows Server Reboot Loop After Installing January 2022 Security Updates

It seems all my blog posts are about Microsoft update failures lately: (

I've seen several reports of Windows Server 2012 R2, 2019, and 2022 getting stuck in a reboot loop after installing the January Windows Updates. Specifically, these updates:

• KB5009624 for Windows Server 2012 R2
• KB5009557 for Windows Server 2019
• KB5009555 for Windows Server 2022

Microsoft is currently aware of the issue.

To fix the issue, restart the computer in Safe Mode which will allow you to login and remove the offending update from Windows Update. You can normally get into Safe Mode by pressing F8 immediately after the server starts.

Domain Controllers are a little more tricky, since there isn't a local user account to login with. For DCs you should restart in Safe Mode with Networking. This will allow you to login with a Domain Admin account.

To remove the update from the command line, run the the appropriate command for your operating system:

Windows Server 2012 R2:

wusa /uninstall /kb:5009624

Windows Server 2019:

wusa /uninstall /kb:5009557

Windows Server 2022:

wusa /uninstall /kb:5009555

I found that if the server is configured to automatically download and install updates it will reinstall the errant update all over again. Grrrr. To prevent this, you can hide the update from reinstalling.

• Uninstall the update and then run run Check for Updates from Windows Update in the server.
• Right-click the update and select Hide Update to prevent it from being reinstalled.

I, for one, am really getting tired of poor quality of updates coming from Microsoft these days. There's simply no excuse for this.

UPDATE - January 17, 2022

Microsoft is releasing Out-of-band (OOB) updates today, January 17, 2022, for some versions of Windows. This update addresses issues related to VPN connectivity, Windows Server Domain Controllers restarting, Virtual Machines start failures, and ReFS-formatted removable media failing to mount. All updates are available on the Microsoft Update Catalog, and some are also available on Windows Update as an optional update. Check the release notes for your version of Windows for more information. Updates for the following Windows versions are available on Windows Update as an optional update. For instructions, see the KB for your OS listed below: ·         Windows 11, version 21H1 (original release): KB5010795 ·         Windows Server 2022: KB5010796 ·         Windows 10, version 21H2: KB5010793 ·         Windows 10, version 21H1: KB5010793 ·         Windows 10, version 20H2, Windows Server, version 20H2: KB5010793 ·         Windows 10, version 20H1, Windows Server, version 20H1: KB5010793 ·         Windows 10, version 1909, Windows Server, version 1909: KB5010792 ·         Windows 10, version 1607, Windows Server 2016: KB5010790 ·         Windows 10, version 1507: KB5010789 ·         Windows 7 SP1: KB5010798 ·         Windows Server 2008 SP2: KB5010799 Updates for the following Windows versions are available only on Microsoft Update Catalog. For instructions, see the KB for your OS listed below: ·         Windows 8.1, Windows Server 2012 R2: KB5010794 ·         Windows Server 2012: KB5010797

 

Read more ...

AAD Connect 2.0.88.0 breaks Shared Mailboxes for Exchange hybrid customers

Microsoft released Azure AD Connect version 2.0.88.0 as a download-only version on 12/16/2021. This update includes several bug fixes and introduces support for syncing AD objects from a single forest to multiple tenants.

However, this version contains a new potentially devastating bug that removes disabled user accounts in AD from Azure AD. 

Because shared mailboxes use disabled user accounts this means those mail users are also deleted from Exchange Online. Cloud users will no longer see on-prem shared mailboxes in the GAL or be able to access them. Inbound mail flow will also be affected for these mailboxes since they no longer exist from an Exchange Online Protection perspective.

Luckily, version 2.0.88.0 is not being pushed as an auto upgrade version, so only customers who download and install it are affected.

AAD Connect version 2.0.89.0 has been released. If you are affected by this bug, you should update to the latest version. See my other updates below.

The workaround is to remove AAD Connect 2.0.88.0 and reinstall the previous AAD Connect version 2.0.28.0. Since Microsoft removes all but the most current version, I've made AAD Connect 2.0.28.0 available on my blog here.

I recommend exporting your current AAD Connect configuration first, then importing it when installing version 2.0.28.0. Be sure to uncheck the "Enable staging mode" when completing the installation. 

During the first sync you will see that the disabled accounts in AD are being synced again to Azure AD.

Update #1 - Dec 22, 2021

The Azure AD Connect Version History website was updated yesterday after my blog post to say that version 2.0.89.0 has been released which addresses this issue. However, at this time only 2.0.88.0 is still available from the AAD Connect download website.

Update #2 - Dec 22, 2021

AAD Connect version 2.0.89.0 is now available for download. Strangely, this new version is no longer listed in the version history. :-/ 

I  have confirmed that the bug has been squashed.

Read more ...

November 2021 Windows Security Updates break OWA published with Azure App Proxy

If you use Azure App Proxy to publish Outlook Web App (OWA) your may find that it suddenly stopped working. This is due to a bug in recent Windows security updates that affects Kerberos delegation.

Microsoft quietly announced this in the Microsoft 365 Message Center as announcement #2750 - Take action: Out-of-band update to address authentication issues on DCs relating to Kerberos delegation scenarios.

There are separate out-of-band updates for all versions of Windows Server from Windows Server 2008 SP2 through Windows Server 2019. Make sure to download the correct update for your version of Windows Server.

At a minimum, your should apply these updates to all the Domain Controllers that reside in the same AD site as your Exchange Servers. The OOB update requires a restart of the DCs where it is applied.

Once installed, OWA published through AAD App Proxy will start working again.

Publishing OWA through Azure App Proxy allows your organization to use Conditional Access and MFA for OWA access. If you would like help with this for your organization, please contact EXPTA Consulting.

Read more ...

Azure AD Connect V1.X versions no longer support the V2 endpoint

Microsoft introduced the Azure AD Connect sync V2 endpoint with version 1.6.4.0 in March 2021. Among the improvements, the V2 endpoint includes performance improvements and allows for synchronization of groups with up to 250K members. Enterprise customers with groups of 50K or more were encouraged to move to the new V2 endpoint.

AAD Connect version 2.0.3.0 was released in July 2021 and was a major upgrade. It supports the V2 endpoint by default, but requires Windows Server 2016 or 2019 due to it's dependency on SQL Server Express 2019 for localDB. There are still many customers running AADC V1.x for this reason.

Today, Microsoft updated the AADC version history to say that the V2 endpoint is no longer available for V1.x versions. 

UPDATE - 11/10/2021: Microsoft just added the following information to the AAD Connect version history:

Known Issues

There is an issue where customers who have the V2 endpoint running with an older version and try to upgrade to a newer V1.6 release will see that the 50K limitation on group membership is reinstated. We will not fix this issue in V1.6 and require customers to upgrade to AADConnect V2.0 if this is an issue for them.

Azure AD Connect V1.x customers are strongly encouraged to update to V2.x, keeping in mind that this may require installing AADC V2.x on a new Windows 2016 or Windows 2019 server. I wrote a step-by-step article on upgrading here.

In the meantime, if you are still using Azure AD Connect 1.x you should make sure you're using the V1 endpoint using the following steps.

First, check to see which sync endpoint you're using with these cmdlets, run from the server where Azure AD Connect 1.x is running:

• Import-Module 'C:\Program Files\Microsoft Azure AD Sync\Extensions\AADConnector.psm1'
• Get-ADSyncAADConnectorExportApiVersion
• Get-ADSyncAADConnectorImportApiVersion

If both Get-* cmdlets return the value "1", you're using the V1 endpoint. Nothing more to do here, except plan to upgrade to AAD Connect V2.x as soon as reasonable.

If the values returned are "2", you're using the V2 endpoint and need to change it back to V1.

• Import-Module 'C:\Program Files\Microsoft Azure AD Sync\Extensions\AADConnector.psm1'
• Set-ADSyncScheduler -SyncCycleEnabled $false
• Set-ADSyncAADConnectorExportApiVersion 1
• Set-ADSyncAADConnectorImportApiVersion 1
• Set-ADSyncScheduler -SyncCycleEnabled $true

Be aware that the V1 endpoint cannot sync groups with 50K+ members. You should plan to upgrade to AAD Connect V2.x as soon as possible.

Read more ...

CHANGE LOG: View Quarantine add-in for Outlook

View Quarantine add-in for Outlook Change Log

---

Please see How to install an Outlook add-in to view the Microsoft 365 End-User Quarantine for a full description and installation instructions.

---

UPDATE #1: Microsoft won't certify my add-in because they say it "does not provide significant value or benefits to commercial marketplace customers". I think most of you will disagree. I'll keep trying to get it certified, but in the meantime you can always install it from my website using the procedures from my blog article.

UPDATE #2 [version 1.0.1.0]: I added automatic localization for 37 languages. Reinstall the add-in to if you need one of these languages. Please let me know if my translations need adjustments.

Read more ...