Basic mail flow troubleshooting in Office 365

In this article I will talk about four major mail delivery scenarios in an Office 365 environment.

A. 3rd Party to Office 365
B. 3rd Party to Hybrid
C. Office 365 to 3rd Party
D. Hybrid to 3rd Party

Before proceeding with the mail delivery scenarios in an Office 365 environment, I would like to summarize the mail delivery flow in a general scenario.

Take for example two domains called fabrikam.com and contoso.com in separate organizations.
User@fabrikam.com is hosted by the email server called server.fabrikam.com and user@contoso.com is hosted by server.contoso.com

The mail flow will be as per below:

1. user@fabrikam.com is logged in to his email client (Web Mail Interface, Outlook, Mozilla Thunderbird etc.)
2. user@fabrikam.com is writing the email into his client for user@contoso.com
3. When user@fabrikam.com is pressing the SEND button, he will send the email to his Email Server, server.fabrikam.com
4. The email is located now in the Message Queue of server.fabrikam.com, which will try to communicate with server.contoso.com in order to deliver the email.
5. server.contoso.com will now make the decision to Accept, Time Out OR to Drop the connection.
6. If server.contoso.com will make the decision to Drop or Time Out the connection, it will inform server.fabrikam.com of this decision.

It will do this through a DSN (delivery status notification) message.

7. This DSN message will be seen in the SMTP Send Logs of server.fabrikam.com.
8. It is now the responsibility of server.fabrikam.com to inform his user, user@fabrikam.com, that it was not able to deliver the message.

It will do this through an NDR (non-delivery report) message.

More info related to NDRs and DSN can be found in the below article

https://technet.microsoft.com/en-us/library/bb232118(v=exchg.150).aspx

9. Once user@fabrikam.com receives an NDR stating that his server was not able to deliver the message, he should inform his administrator to check the server logs and see why the message was not sent.

• Is it due to connection issues on his side?
• Is his IP listed on third-party blacklists?
• Was there an issue with his server at that moment? Etc.

Let’s return now to the mail flow delivery scenarios in an Office 365 environment. I will start with the simple one first.

Appendix:

O365 – Office 365

EOP – Exchange Online Protection

EXO – Exchange Online
 
A. Contoso.com is hosted in Office 365 and the MX is pointing to Exchange Online Protection

• user@fabrikam.com is trying to send an email to user@contoso.com
• Contoso.com MX is contoso-com.mail.protection.outlook.com
• Contoso-com.mail.protection.outlook.com is the entry point in EOP
• user@fabrikam.com will send an email from his server called server.fabrikam.com

1. user@fabrikam.com is logged in to his email client (Web Mail Interface, Outlook, Mozilla Thunderbird etc.)
2. user@fabrikam.com is writing the email into his email client for user@contoso.com
3. user@fabrikam.com presses the SEND button and sends the email to server.fabrikam.com.
4. The email is located now in Message Queue of server.fabrikam.com, which will try to communicate with EOP in order to send the email.

Please note that Exchange Online Protection has a predefined ranges of IPs.

All Office 365 customer’s MX records are using the IPs from those ranges.

It will be almost impossible to provide for the millions of Office 365 customers a unique IPv4.

So how does EOP make the decision to route the emails to the correct customers?

5. How the email will be delivered to contoso.com is up to server.fabrikam.com

• Through DNS resolution and look up of the MX record of contoso.com
• Through a send connector based on certificate
• Through a basic send connector using contoso-com.mail.protection.outlook.com as smart host

Please note that the delivery settings on server.fabrikam.com will need to be the same on EOP as well.

For example, if server.fabrikam.com is sending the emails to contoso.com through a connector based certificate, then in EOP an Inbound Connector must exist to match the exact same settings.

In a daily scenario, the email will be send through DNS resolution and look up of the MX record of contoso.com

• Do I accept messages from that IP?
• Do I have a connector with that certificate? If Yes, in which tenant?
• Do I have a connector with that IP? If Yes, in which tenant?
• Do I accept messages from user@fabrikam.com?
• Do I have a recipient user@contoso.com? If Yes, in which tenant?

How does an Office 365 customer know if there is an issue with the sending server or with Office 365?

Troubleshooting

1. Perform a message trace in O365 with user@contoso.com as a recipient, by following one of the below articles:

Perform message trace in Office 365

Find an fix email delivery issues in O365

2. If you cannot see traces of the emails, then server.fabrikam.com had issues connecting to EOP.
3. Check if the Public IP of server.fabrikam.com is not on the EOP 3^rd Party blocking lists.

Check the EOP 3^rd Party blocking lists

4. In EOP, create an Inbound Connector of type Partner and add the Public IP of server.fabrikam.com as the sending server.

• Access the below article:

https://technet.microsoft.com/en-us/library/dn751021(v=exchg.150).aspx#setupconnectorfrompartner

• Follow the steps from:

Example 4: Require that all email sent to your organization from the Internet is sent from a specific IP address (third-party email service scenario)

The purpose of creating an Inbound Connector is to make sure that the IP of server.fabrikam.com is not subject to EOP throttling.

What is EOP throttling?

 
Please note that if the messages are not visible in the message trace, it is impossible to pursue an investigation without the Send SMTP Logs from server.fabrikam.com.

 
 
B. Hybrid deployment, Contoso.com is hosted both in Office 365 and on the on premises Exchange. The MX is pointing to the on premises Exchange.

• The MX of contoso.com is pointing to mail.contoso.com
• user@contoso.com was migrated to Office 365
• Remote routing address after the migration is user@contoso.mail.onmicrosoft.com
• user@fabrikam.com is hosted by the 3^rd Party email server server.fabrikam.com
• user@fabrikam.com is trying to send an email to user@contoso.com

What is a remote routing address?

1. user@fabrikam.com is logged into his email client (Web Mail Interface, Outlook, Mozilla Thunderbird etc.)
2. user@fabrikam.com is writing the email into his email client for user@contoso.com
3. user@fabrikam.com presses the SEND button and sends the email to server.fabrikam.com.
4. The email is located now in Message Queue of server.fabrikam.com, which will try to communicate with mail.contoso.com in order to send the email.
5. How the email will be delivered to contoso.com is up to server.fabrikam.com

• Through DNS resolution and look up of contoso.com MX record
• Through a connector based on certificate
• Through a basic connector using mail.contoso.com as a smart host

Please note that the delivery settings on server.fabrikam.com will need to be the same on mail.contoso.com as well.

Example if server.fabrikam.com is sending the emails to contoso.com, through a connector based certificate, then on mail.contoso.com, a Receive Connector should exist to match these exact settings.

In a daily scenario, the email will be sent through DNS resolution and look up of the MX record of contoso.com

• Do I accept messages from fabrikam.com or the IP of server.fabrikam.com?
• Do I have a connector for this fabrikam.com? If yes, what type?
• Do I accept messages from user@fabrikam.com?
• Do I have a recipient user@contoso.com?

The DSN will be visible in the SMTP Send logs of server.fabrikam.com and also in the Receive SMTP Logs of mail.contoso.com.

• Is it a User Mailbox? If yes, on which mailbox database and server is the mailbox located?
• Is it a Mail User? If yes, what is the External Email Address?
• Is it a Mail Contact? If yes, what is the External Email Address?
• Is it a Remote Mailbox? If yes, what is the Remote Routing Address?
• Is a Mail Enabled Public Folder? If yes, where is it located?

 
So how will the email be delivered to Office 365 then?

• Is it a User Mailbox? If yes, on which mailbox database and server is the mailbox located?
• Is it a Mail User? If yes, what is the External Email Address?
• Is it a Mail Contact? If yes, what is the External Email Address?
• Is it a Remote Mailbox? If yes, what is the Remote Routing Address?
• Is a Mail Enabled Public Folder? If yes, where is it located?

 
How does an Office 365 customer know if there is an issue with the sending server, his on-premises server or with Office 365?

Troubleshooting

1. Perform a message trace in O365 having user@contoso.com or user@contoso.mail.onmicrosoft.com as recipients, by following one of the below articles.

Perform message trace in Office 365

Find an fix email delivery issues in O365

2. If you cannot see traces of the emails, then continue with the next steps.
3. Perform a message trace on mail.contoso.com having user@contoso.com as a recipient, by following the below articles:

Perform a message trace in Exchange 2016

Perform a message trace in Exchange 2010

4. If you cannot see traces of the emails, then server.fabrikam.com had issues connecting to mail.contoso.com. Proceed with the next steps.
5. Check mail.contoso.com SMTP Receive Logs to see if it has rejected the connection.

Please note that SMTP Logs are not enabled by default in Exchange. They must be enabled on each Receive or Send connector.

You can follow the below articles on how to enable them and where to find them.

Check the Exchange 2010 Receive SMTP Logs

Check the Exchange 2010 Send SMTP Logs

Check the Exchange 2013 SMTP Logs

6. If you cannot see connection attempts from server.fabrikam.com, then either you have a Firewall issue or server.fabrikam.com didn’t attempt to deliver the email at all.
7. If you see traces of the emails, then mail.contoso.com must have had issues connecting to EOP. Proceed with the next steps.
8. Check your Exchange Message Queue by following the below articles.

Check Exchange 2016 message queue

Check Exchange 2010 message queue

9. You should pay attention to:

• Check if the messages for user@contoso.mail.onmicrosoft.com are going through the proper Hybrid Send Connector
• The failure message on the Queue or Message. Usually you should see TimeOut or Drop.

More details about this behavior

Check the EOP 3rd Party blocking lists

 
Please note that if the messages are visible in the Message Queue of mail.contoso.com, but cannot be delivered to Office 365, it is impossible to pursue an investigation without the Send SMTP Logs from mail.contoso.com.

 
C. Contoso.com is hosted in Office 365 and user@contoso.com sends an email to 3^rd Party user@fabrikam.com

• Contoso.com sends the email directly and not using a dedicated Outbound Connector
• user@fabrikam.com has the mailbox hosted on 3^rd Party server called server.fabrikam.com
• MX for fabrikam.com is pointing to server.fabrikam.com

I will not focus on the email delivery steps, because they are somehow similar to the ones from Scenario A but reversed.

I will however insist on the troubleshooting done to determine where the issue is.

So how does an Office 365 customer know if there is an issue with the recipient’s server or with Office 365?

Troubleshooting

1. Perform a message trace in O365 with user@contoso.com as the sender, by following one of the below articles

Perform message trace in Office 365

Find an fix email delivery issues in O365

2. Check the status of the message.
3. Is it Pending, Defer, Failed?
4. If Yes, double click on the message and you should see a message like the one below:

• In this case the email failed because Yahoo rejected my message.
• As we can see, the email is sent from Office 365 without being marked as Malware or FilteredAsSpam.
• So we can assume that EOP marked it as safe and it was sent through the normal pools of IPs and not through HRDP (Higher Risk Delivery Pool).

What is HRDP?

• If the email would have been sent through the HRDP, then we could have considered that is possible that the HRDP IPs are listed on an external blocking list used by server.fabrikam.com

 
Please note that the email could have been Delivered to server.fabrikam.com, but internally the server could have rejected it and choose not to send an NDR back.

 
How can I do this?

1. For beginners, you can download and open telnet25 from https://telnet25.codeplex.com/
2. Open telnet25.exe and add the fields below:


 

3. Click on Telnet tab and then click on SEND button


 

4. For successful connection you should see something like this:

 
220 server.fabrikam.com

EHLO contoso.com
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-8BITMIME
250-BINARYMIME
250 CHUNKING

mail from: user@contoso.com
250 2.1.0 Sender OK

rcpt to: user@fabrikam.com
250 2.1.5 Recipient OK

DATA
354 Start mail input; end with .


5. For unsuccessful connection, please check what step is failing and what is the message.

Please note that it’s possible that TELNET does not work, because it can be blocked by the internal network of server.fabrikam.com

• SPF record
• DKIM
• DMARC (If it’s the case)

 
So, all this being said…
 
Please note that if the email appears as Delivered and no NDR is received, it is impossible to pursue an investigation from Office 365 side.
The administrator of user@contoso.com should contact user@fabrikam.com and troubleshoot internally.

 
D. Hybrid deployment. Contoso.com is hosted both in Office 365 and on-premises Exchange and has Centralized Mail Flow enabled.

• Contoso.com has an Hybrid Outbound Connector using a smart host mail.contoso.com
• user@contoso.com was migrated to Office 365
• Remote routing address after the migration is user@contoso.mail.onmicrosoft.com
• user@fabrikam.com has the mailbox hosted on 3^rd Party server called server.fabrikam.com
• MX for fabrikam.com is pointing to server.fabrikam.com
• user@contoso.com is trying to send an email to user@fabrikam.com

What is Centralized Mail Flow?

Centralized Mail Flow means that all emails sent from Office 365 internal users to external recipients will be sent out through the on-premises Exchange Server.

1. user@contoso.com is logged in to his email client (OWA, Outlook, Mozilla Thunderbird etc.)
2. user@contoso.com is writing the email into his email client for user@fabrikam.com
3. user@contoso.com presses the SEND button and sends the email to Office 365 Transport Servers.
4. The email is located now in then Message Queue of the Office 365 Transport Servers, which will see that there is a Hybrid Outbound Connector having Centralized Mail Flow enabled.
5. Based on these settings, Office 365 will try to communicate with mail.contoso.com to send the email.
6. How the email will be delivered to mail.contoso.com is based on the settings on the Hybrid Outbound Connector from EOP.

• Through a connector based on certificate
• Through a basic connector using just mail.contoso.com as a smart host

Please note that the delivery settings on the Hybrid Outbound Connector must be the same on the Hybrid Receive Connector of mail.contoso.com.

For example, if EOP is sending the emails through a connector based certificate, then on mail.contoso.com a Receive Connector must exist to match the exact same settings.

• Do I accept messages from that EOP IP?
• Do I have a connector with that certificate?
• Do I have a connector with that IP?
• Do I have a recipient user@contoso.com?

• Through DNS resolution and look up of fabrikam.com MX record
• Through a connector based on certificate
• Through a basic connector using server.fabrikam.com as a smart host

And then the same steps we discussed above will happen. You get the idea

 
How does an Office 365 customer know if there is an issue with the recipient server, his on-premises server or with Office 365?
 
Troubleshooting

1. Perform a message trace in O365 with user@contoso.com as the sender, by following one of the below articles

Perform message trace in Office 365

Find an fix email delivery issues in O365

2. Check the status of the message.
3. Is it Pending, Defer, Failed?
4. If Yes, double click on the message and you should see a message like the one below:

• In this case the email failed because Yahoo rejected my message.
• As we can see, the email is sent from Office 365 without being blocked as Malware
• You should pay attention to the fact that the sending attempt is being made to mail.contoso.com
• Also pay attention to the name of the Outbound Connector, it should have the same name as you see in O365.
• In this scenario, I would expect the email to be sent through the Hybrid Outbound Connector to mail.contoso.com

 
Example of SEND event through the Hybrid Connector.
:ExternalSendLatency=1.949;S:Microsoft.Exchange.Hygiene.TenantOutboundConnectorCustomData=Name= Name of the Outbound Connector;ConnectorType=OnPremises;UseMxRecord=False’;
S:OutboundProxyFrontEndName=HE1EUR01FT007;S:IsSmtpResponseFromExternalServer=True;S:Oorg=contoso.com

Again, you should pay attention to the fact that the sending attempt is being made to mail.contoso.comthrough the Hybrid Outbound Connector.

Please note that SMTP Logs are not enabled by default in Exchange. They must be enabled on each Receive or Send connector.

You can follow the below articles on how to enable them and where to find them.

Check the Exchange 2010 Receive SMTP Logs

Check the Exchange 2010 Send SMTP Logs

Check the Exchange 2013 SMTP Logs

Please check the Office 365 IPs and URLs list and update it on your Firewall.

However, all the above steps will be double-checked and if the issue is not with O365, you will be asked to investigate internally.
 
So, all this being said…
 
Please note that if the email appears as Delivered and no NDR is received, an investigation from Office 365 side will be done in order to determine if the Hybrid settings on Office 365 and mail.contoso.com are correct.
If they are, the administrator of user@contoso.com needs to troubleshoot internally.