Lync, Office365, Skype4B, Teams

Despite the major announcements around Microsoft’s change to position Teams as the first-and-foremost UC solution in the portfolio over Skype4B, the other just-as-big-change was the announcement that the underlying network infrastructure used to support Skype4B within Office365 was changing to introduce TRAP (aka – Transport Relay).  Despite my love for creating in-the-weeds technical posts, I’ll defer on reinventing that wheel since there are already sufficient resources out there in the blog-o-sphere that talk about TRAP:

What I will choose to cover, however, is what those TRAP IP addresses are and how they have changed over the past few months.

TRAP at the Start

To quote Admiral Ackbar, “It’s a Trap!”…  A “small” but worthy goal of moving towards AnyCast routing for Skype4B so that media performance is increased by getting traffic onto the high-performance Microsoft network as soon as possible.  It was a simple beginning that only consisted of just over 10 IP addresses.  Microsoft didn’t explicitly publish the IP addresses, but you could obtain bits and pieces of the IP addresses in play if you knew where to look.  For instance, if you examined the Wireshark traces of the original Skype for Business Network Assessment Tool, you could see that the 13.107.8.2 Global Relay IP address widely advertised on the Internet wasn’t actually where the media traffic ended up coming to/from:

At that time, RTP streams were actually flowing to/from a 104.44.195.154 IP address instead of the global Anycast IP address.  Even for the third-party partners such as IR Prognosis, you would see the global AnyCast IP was only part of the puzzle used for the tests:

104.44.195.X
104.44.195.X
104.44.195.X
104.44.200.X
104.44.200.X
104.44.200.X
104.44.200.X
104.44.201.X
104.44.201.X
104.44.201.X
13.107.64.X
13.107.65.X
13.107.8.2

Sadly, the only way I orginally saw the list above is because IP Prognosis’s tool had them listed in the tool portal.  If it hadn’t been for that chance viewing, I don’t believe I would have been able to see them all because I never saw them listed anywhere else.  Even so, it was relatively clear that most production Skype4B traffic in Office365 wasn’t actually traversing the testing IPs and that this was a pre-cursor to a big change from Microsoft….it just wasn’t completely clear to everyone at the time what was coming:

I had suspicion that AnyCast was coming, but Microsoft’s tight-lipped-nature didn’t expose much.  Ignite 2017 changed that reality and they’ve gone full blown with AnyCast/TRAP.  Even so, the announcement was that tenants would be enabled in the future, which leaves many to wonder when this will occur and how this really changes things for them.

TRAP in CY 2018

Microsoft still does not officially publish the TRAP IP addresses for customers to examine, which is a bummer.  Even though they aren’t officially published, there is a deceptively simple way to obtain a new’ish list, which contains significant changes that goes to show it truly has begun to roll out to wider audiences.  Obtaining this list is far simpler than before, thankfully, and again comes thanks to the Skype for Business Network Assessment Tool.  The testing tool was updated in late 2017 and that update contained some new functionality, one of which included a “/connectivitycheck” parameter.  If you add on a final “/verbose” parameter…voila!:

.\NetworkAssessmentTool.exe /connectivitycheck /verbose

If you pass all the results to “| clip” and drop it in Excel, you can eventually filter out the following list of 56 unique IP addresses:

104.44.195.1
104.44.195.11
104.44.195.254
104.44.200.1
104.44.200.254
104.44.200.53
104.44.200.92
104.44.201.1
104.44.201.146
104.44.201.254
13.107.64.2
13.107.65.5
13.107.8.2
52.114.124.1
52.114.124.108
52.114.124.110
52.114.124.221
52.114.124.254
52.114.125.1
52.114.125.254
52.114.126.1
52.114.126.254
52.114.127.1
52.114.127.254
52.114.188.1
52.114.188.18
52.114.188.19
52.114.188.21
52.114.188.254
52.114.188.27
52.114.188.28
52.114.189.1
52.114.189.254
52.114.190.1
52.114.190.254
52.114.191.1
52.114.191.254
52.114.220.1
52.114.220.10
52.114.220.254
52.114.221.1
52.114.221.254
52.114.222.1
52.114.222.254
52.114.223.1
52.114.223.254
52.114.60.1
52.114.60.254
52.114.60.32
52.114.60.33
52.114.61.1
52.114.61.254
52.114.62.1
52.114.62.254
52.114.63.1
52.114.63.254

The list has grown considerably but more important is the fact that it has grown into the 52.112.0.0/12 block of IP addresses that Microsoft explicitly lists as used within Skype for Business Online.  If your break out your CIDR skills and crunch the numbers, you’ll see that that range goes from 52.112.0.0 to 52.115.255.255, so it’s clear that Microsoft has begun to move the technology into the production Office365 IP ranges.  I say “production IP ranges” because most of us involved with Skype4B Online implementations know that the IP consolidation Microsoft has been performing results in a very large percentage of tenants being located in the 52.112.0.0/12 range.  If you perform a trace-route, you’ll see that these new IP addresses are given PTR DNS records under the .relay.teams.microsoft.com domain:

52-114-124-1.relay.teams.microsoft.com

52-114-60-1.relay.teams.microsoft.com

52-114-189-1.relay.teams.microsoft.com

As you can see from my results, the RTTs are a bit over the map.  If these truly are AnyCast IP addresses, then Comcast should simply be peering via BGP to the closest peer router advertising AS8075 (and thus keeping the RTT low), but you’ll notice slight routing differences between the examples.  Assuming that these are AnyCast IP addresses, then I would expect the RTT for the final hop to be a bit lower but it is possible that some of the 52.X.X.X IP addresses in the testing list are used as geographically-regionalized relays, such as NOAM, SOAM, EMEA, APAC, etc.  For instance, 52.114.60.1 seems to traverse Los Angeles (be-3-0-ibr02-laxo2 and ae72-0-lax) and Tokyo (ae26-0-tya and ae-10-0-tyo01) before it arrives at its final destination.

Note: I don’t actually know for certain that those acronyms correlate to those cities.  It is an absolute possibility that I could be wrong, but examination of the naming standard that Microsoft uses (based off other trace-route tests and from the Ignite 2017 content) seems to suggest that this routing path is likely.

If that is actually the case (geographically-regionalized relays), then it would be expected that my trace-routes have higher-than-expected RTT since the data packets must traverse a longer distance to actually arrive to the node in the remote locale.  Even so, all those trace-routes show that my client is able to ingress into the Microsoft network in roughly 20 milliseconds via an edge node in Atlanta, GA (ae3-0-atb and ae8-0-atb), which means that BGP peering is working exactly as it should in taking my data packets from Nashville, TN and allowing them to ingress at the closest Microsoft network ingress location in Atlanta, GA:

52-114-124-1.relay.teams.microsoft.com

52-114-60-1.relay.teams.microsoft.com

Finding Peering Locations

There are several places that Microsoft puts information in relation to Office365, Azure, and its global network, but they don’t make it particularly easy to find and they don’t include everything in a single one-piece-gift-with-a-bow-package.  As a result, you have to dig through several pieces of information to pull out what exactly you may be looking for.  For example…

Office365 Datacenter Locations

If you are looking for where Microsoft’s Office365 data centers are and what services are offered from each, this is an important URL to bookmark.  Even so, it does not make any mention of the Microsoft network ingress locations.

Azure CDN POP Locations

If you are looking for where the Azure Content Delivery Network Points-of-Presence are, this is an important URL to bookmark.  While Office365 does utilize some of the same CDN infrastructure as Azure, it isn’t a complete 1:1 reference and as a result it doesn’t completely match up with the Microsoft network ingress locations.

Azure Regions

If you are looking for where Microsoft’s Azure data centers are and what services are offered from each, this is an important URL to bookmark.  Even so, it does not make any mention of the Microsoft network ingress locations.

Ignite 2017 Content – BRK1005

Probably the best content on listing network ingress points comes from an Ignite 2017 presentation.  Slide 8 of the BRK1005 presentation includes a table with 69 of the peering locations – even though more than 69 are actually available – and it also includes a hyperlink to the most important piece of information…the Microsoft network Autonomous System number.

This particular Ignite 2017 session is probably the most important for network architects/engineers to watch because it truly does provide the most in-depth information about how Microsoft is expanding and enhancing their networks supporting Office365.

PeeringDB

This is probably the single most important list out there because it includes all ISPs (public and private) and geographic locations where Microsoft peers their Autonomous System Number 8075 with the Internet.  If you are looking for whether or not your office location has a peer available in-region and you cannot find it on any of the other resources, the PeeringDB list is the authoritative resource that may tell you.  You may typically filter the list by entering the name of the region/city (such as “Hong Kong” or “Brazil” or “Atlanta”) but you may have to dig a bit deeper and actually enter your ISP name if all else fails:

Hong Kong

Brazil

Atlanta

Remember that while Microsoft may advertise the ASN number from these locations, it is up to each ISP in those regions to receive and honor the BGP routes that Microsoft provides.  If you find traffic routing across other carriers or potentially arriving at an ingress node that isn’t in-region, you’ll need to reach out to your ISP & Microsoft so that they can hash out the issues with peering (assuming there is an in-region Microsoft node).  For some locations – like China – all traffic should ingress via Hong Kong so that particular routing path is not cause for alarm because it is by-design.  For most large regions in NOAM, SOAM, EMEA, and APAC, Microsoft edge nodes are available in-region and it should greatly reduce the distance required to hop on their high-speed network.

Final Notes

In any event, the migration to TRAP is very much real and Microsoft’s guidance to “drop traffic on the Internet as close as possible to the user” aims to take advantage of this network infrastructure.  This is also why Microsoft wants customers to use the Internet and not ExpressRoute for Office365, because ExpressRoute circuits are effectively centralized Internet egress points which don’t offer much advantage to users unless an ExpressRoute circuit is available at every single physical location that exists within an enterprise.  Only the largest enterprises with the biggest checkbooks could afford that type of infrastructure, so as a result most enterprises would have ER circuits only at their major data center locations, thus negating the benefits of AnyCast routing for users that may connect to those data centers from far-away geographic regions.  Microsoft will continue to enhance their network infrastructure and it’s up to ISPs and Enterprises to make sure that routing to the AS8075 network is as fast and performant as possible.