Where Does Exchange Online Message Tracking Data Come From?

The origin and nature of the data used by Promodag Reports for its Office 365 mailbox traffic reports are the subject of recurring questions. Where does this data come from? What information does it contain? What are its limitations? How is it collected by Promodag Reports? Let's take a look at some of these questions.

Message trace in the Exchange Admin Center and PowerShell

Origin and nature of available data

As Exchange Online is a hosted environment, it is not possible to import physical files (or tracking logs), as we have been doing for over two decades in on- premises Exchange environments. But Microsoft does allow access to message tracking data through the message trace feature in the Exchange Administration Center and, under the hood, through the PowerShell command Get-MessageTrace. 

The data returned by this command will seem familiar to Exchange administrators accustomed to browsing the contents of Exchange's on-premises message tracking files: indeed, it's almost identical: sender address, recipients, size, subject, Message ID etc.

Data limitation compared to On-Premises Exchange

The most important limitation is the data retention period: this is limited to 10 days with no possibility of modification, whereas the default (and modifiable) period was 30 days for On-Premises Exchange. Once these 10 days have elapsed, message tracking data disappears for good.

Besides, some of the information previously available has changed, or even disappeared:

• The recipient level of sent messages (To, Cc, Bcc),
• The address of the actual sender of a message sent from a delegated mailbox.

This of course has an impact on the level of detail in the traffic reports. The Find Specific Messages report therefore displays all recipients as if they were the main recipients of the message sent, and messages sent from a delegated mailbox are identified as having been sent by it in all traffic reports. We have, however, created a report that partially circumvents the latter problem: Delegated Mailings, which was the subject of this post in 2023.

Exchange Online data processing by Promodag

Promodag has developed two distinct tools: StoreLog, a freeware package whose scope is limited to generating Exchange Online tracking logs and importing them into a database, and Promodag Reports, a much more elaborate fee-based solution enabling the creation of multiple tabular and graphical reports.

With Promodag StoreLog 

To fully understand the challenge of exploiting this data, we refer you to this post we published in 2014 when developing a method for our free StoreLog utility. The data returned by PowerShell is sorted in anti-chronological order and by pages of 5,000 messages, which may overlap if the command has been run several times in succession. 

We have therefore created an automatable process to retrieve the data in hourly files, sorting them in chronological order and in columns whose content is as close as possible to that of Exchange's message tracking files. The algorithm also checks for consistency, thus avoiding the creation of duplicates.

With Promodag Reports 

Generating Exchange Online message tracking was integrated into our flagship reporting tool in 2014 and has been constantly improved ever since.

Thus, the Create Office 365 message tracking files option allows you to manually generate 24 hourly Office 365 tracking logs per day, going back 10 days to today's date one hour ago. This process can be entirely automated.

By default, these tracking logs are created in the C:\Users\PublicDocuments\Promodag\Reports\Office365MessageTracking<name of your tenant> directory.

The resulting files are named according to the following convention: 

MSGTRKC <date in YYYYMMDD format> number to follow from 00 to 23-1.log.

File number 00 contains events from midnight to 01:00 UTC, and file number 23 those from 23:00 to midnight UTC.

Thus, the tracking log named MSGTRKC20240920-1.log will contain the traffic events of September 20, 2024 between 8:00 PM and 9:00 PM UTC, i.e. between 2:00 PM and 3:00 PM Eastern Time (USA & Canada).

What's the point of creating Exchange Online tracking logs?

So why use this method and go to all this trouble? For two reasons:

• Firstly, because the availability of this data in the Microsoft 365 Cloud is very short, so it's more than worthwhile to perpetuate it by creating files that can be read and imported at any time. This enables reporting at the message subject level over very long periods, unlike native Microsoft 365 Administration Center reports.

• Secondly, because it literally handles Exchange Online (named “Office 365” in the Promodag Reports interface) as if it were an Exchange server producing tracking logs and enables its integration seamlessly for the end-user.

You can download the latest version of Promodag Reports on this page, and that of Promodag StoreLog on this one. Should you have any technical or commercial questions, or simply require further information, please feel free to use our contact form.