MEC 2014 is Right Around the Corner! Are You Registered?

I'm very much looking forward to the Microsoft Exchange Conference March 31-April 2 in Austin, TX. I hope you can join me there!

I'll be moderating three MEC Unplugged interactive sessions this year and will be on the experts panel for the Exchange Deployment session.  It's important to know that the Experts Unplugged sessions will not be recorded, so if you want to see them or participate in the discussion you'll need to register for MEC.

Session	Date/Time	Room	Speakers
Experts Unplugged: Architecture – Client Access and Connectivity	Tuesday, April 1 9:00AM - 10:15AM	MR 17b	Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Venkat Ayyadevara
Experts Unplugged: Architecture - Transport and Hygiene	Tuesday, April 1 10:45AM - 12:00PM	MR 17b	Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry, Wendy Wilkes
Experts Unplugged: Architecture - Transport and Hygiene (repeat session)	Wednesday, April 2 8:30AM - 9:45AM	MR 13ab	Brian Reid, Jeff Guillet, Khushru Irani, Ross Smith IV, Scott Landry, Wendy Wilkes
Experts Unplugged: Exchange Deployment	Wednesday, April 2 8:30AM - 9:45AM	MR 17b	Brian Day, Greg Taylor, Jeff Guillet, Jeff Mealiffe, Ross Smith IV, Scott Schnoll

Click the sessions above to add them to your MEC schedule. If you haven't registered for MEC yet, it's not too late.

There will be no paper copy of the schedule this year. The schedule will be available via an HTML5 app that should work on all platforms, but I suggest you print a copy of your schedule or add it to your calendar before you arrive. Technology sometimes has a nasty way of not working when you need it.

Here's a breakdown of the current attendee profile, based on registered attendees so far.

With 87% of the attendees from the US, it looks like Europe would definitely be served well by having its own MEC. That would better align with the global deployment of Exchange Server. I'd say the chances of seeing MEC in Europe would be pretty slim, though.

Read more ...

Announcing the 7th Annual UC Roundtable at TechEd 2014, Houston!

I'm pleased to announce the 7th Annual UC Roundtable at Microsoft TechEd North America 2014 in Houston, TX!

The purpose of the UC Roundtable is to gather Exchange and Lync admins, MCMs, MVPs, Exchange product group members, architects, and experts for a free-flowing discussion about issues, questions, and experiences related to Exchange, Office 365, and Lync Server.  If you work with Exchange, Office 365, or Lync you need to be here!

The UC Roundtable will be held Wednesday, May 14, 2014 from 6:00-8:00PM CDT and will be within walking distance or a short cab ride from the TechEd hotels. A special thanks to my friends at F5 who will be hosting the event for the third year in a row!

Please RSVP to jeff@expta.com for event details and location. I will email you with the location details once they're set.

Help spread the word and I hope you can join me!

Read more ...

Pending Messages in Exchange Online Will Not Be Delivered

UPDATE: I'm happy to say that the bug described in this article has been squashed. Deferred emails will automatically be recategorized in Office 365 transport within a few hours and will then be delivered.

There’s a pretty big unpublicized bug in Office 365 transport that you need to be aware of.

If emails cannot be delivered due to a problem with TLS those emails will queue in Exchange Online Protection (EOP) and will be marked as Pending or Deferred. If you change the Outbound Connector in EOP to work around the problem these messages will NEVER be delivered, even after the transport issue is resolved. This happens because the outbound messages in Office 365 are stamped with the old TLS configuration and are not reevaluated when the Outbound Connector configuration is changed.

The following example shows a message that was sent at 2:46PM UTC and the status shows as Pending. The last event for this message shows DEFER at 3:29PM UTC with the detail, "The last attempt to deliver the message encountered an error".

Sample Message Trace from Office 365

For example, say the TLS certificate expires on your hybrid server. Inbound messages from EOP to the hybrid server will queue because the Outbound Connector is using Forced TLS, but the certificate is invalid. If you resolve the problem by reconfiguring the Office 365 Outbound Connector to use Opportunistic TLS the problem is solved for new emails - they get delivered right away, but the Pending messages will never get resubmitted and eventually expire after 48 hours.

This same behavior would occur if you have a custom Outbound Connector that forces TLS with a business partner. If their TLS certificate expires or they reconfigure their server to not use TLS. The messages will not be resubmitted to use the new configuration.

Incredibly, there currently is no way for Microsoft to resubmit these messages like there is for on-prem Exchange. After opening a high-priority case with Microsoft Online, the only "solution" they could give is to contact all the senders and ask them to resend their emails.

To work around this, you may want to ensure that your Outbound Connectors that use TLS are configured to use opportunistic TLS. That way if something changes in the TLS configuration, such as the receiving server's cert expires or TLS is disabled, emails can still be delivered.

Read more ...

Troubleshooting TLS SMTP Connections to Exchange Online Protection

By default Office 365 uses Transport Layer Security (TLS) to send encrypted SMTP emails between Exchange Online and Exchange on-prem. This provides end-to-end encryption of emails between your on-prem Exchange Hybrid Server and Exchange Online Protection (EOP), just like they were the same organization.

TLS utilizes x.509 (SSL) certificates to encrypt the email in transport. Exchange Online and Exchange 2013's implementation of TLS differs from previous implementations of TLS in several important ways:

• The certificates used for TLS must be issued by a trusted third-party CA (Digicert, Godaddy, Verisign, etc.)
• The CRL for the certification authority must be available. If Exchange or O365 can't read the CRL it will not trust the certificate.
• The FQDN that the Receive Connector provides in response to EHLO must match the subject name or a subject alternative name on the certificate. SAN certificates and wildcard certificates are both valid for TLS use. If you use a wildcard cert the FQDN used on the connector can be any name that is valid for the wildcard cert's domain.
• In previous versions of TLS any certificate could be used to encrypt SMTP traffic, even expired or self-signed certificates. This is not the case with Office 365 and Exchange 2013.

As previously mentioned, TLS is required by default for SMTP communication between Hybrid servers and Exchange Online Protection (EOP). When you run the Hybrid Configuration Wizard it will configure forced TLS for all Send and Receive Connectors. Note that this configuration is updated whenever you run the HCW, so if you reconfigure the connectors for opportunistic TLS or turn it off completely, the HCW will reconfigure them to use forced TLS again.

The following options are available for TLS encryption in Office 365:

• Force TLS - Email sent across this connector must use TLS. If TLS is unavailable messages will queue until they are delivered or expired.
• Opportunistic TLS - The connector tries to setup a TLS connection using the STARTTLS verb. If a TLS connection cannot be made the connector falls back to regular ESMTP or SMTP. This is functionally equivalent to turning TLS off.

Normally the use of TLS is configured on Receive or Inbound Connector. You do this to control how email is accepted by your domain and it can be easily configured using the Exchange Admin Console. 

Office 365's Inbound Connector from Hybrid Server

Send Connectors can also be configured to require TLS. This is used to enforce that email is sent by this connector must only use TLS, and is configured in the shell using the RequireTLS property:

Outbound Connector TLS Settings to Office 365

Using this configuration if the corresponding Receive Connector does not offer or require TLS, messages will queue on the sending server until they are finally delivered or expire.

You can easily tell if a receiving SMTP server is configured to use TLS using Telnet. Install the Telnet Client feature, Telnet to the server using TCP port 25, and look for the STARTTLS verb after issuing the EHLO domain.com command. For example:

Telnet servername 25

If you see that the STARTTLS verb is missing, the server is not offering TLS. If your Send Connector is configured to require TLS, the messages will queue on the sending server with the error,

451 4.4.0 Primary target IP address responded with: "451 5.7.3 STARTTLS is required to send mail."

There are several reasons that the STARTTLS verb might be missing:

• The receiving server is not configured to Force TLS or use Opportunistic TLS.
• The sending server's IP is on an SMTP block list (aka SMTP blacklist or SMTP blocklist). Office 365 will not attempt to send TLS traffic with a server it can't trust.
• The receiving server is configured to only respond to SMTP (not ESMTP) commands. TLS is part of the ESMTP protocol.
• Your firewall is doing some form of email inspection and is filtering "unsafe" verbs from the SMTP conversation. Some examples of firewalls that do this are:
• Cisco ASA
• Dell SonicWall
• Barracuda

Office 365 uses internal and external SMTP blocklists to protect the network. If the sending server is on one of these black lists the EOP servers will not offer TLS. You can test this by sending a non-TLS email to EOP using Telnet. See Brian Reid's article, "Cannot Send Emails to Office 365 or Exchange Online Protection Using TLS". The response from the EOP server will tell you which block list you're on and how to request removal.

If the receiving server requires TLS, but the sending server is not configured to use a TLS certificate messages will queue on the sending server with the error,

451 4.4.0 Primary target IP address responded with: "451 5.7.3 Must issue a STARTTLS command first."

The fix here is to configure the Send Connector to use TLS and a valid certificate using the Enable-ExchangeCertificate cmdlet, such as:

Enable-ExchangeCertificate -Thumbprint 5DC5902752816FD2FC51D5564C363F68D8F7FFC4 -Services SMTP

Make sure to use Get-ExchangeCertificate to get the correct certificate's thumbprint for the command above.

Hopefully this information will help you understand TLS and will assist you with troubleshooting.

Read more ...