A Glitch in
the Matrix

November 2017

The Internet: A Global Network

Map showing reported Internet issues across the U.S on November 6th, 2017
How a Simple Error Can Cause Chaos
On Monday, November 6th, 2017 many people across the United States experienced issues with IP based services where users experienced slow connection speeds, unreachable websites, disruption to normal communications, and complete service outage. So what happened? It was all caused by a configuration error at Level 3. To understand how it affected so many we need to take a look at what the Internet is.

In simple terms, the Internet is a global network of billions of connected devices and other networked devices. All of these devices are connected via physical cables (some even run under the oceans to connect continents) to form networks. Even wireless connections like Wi-FI and 4G ultimately rely on these physical connections. In this mass of cables and connections your Internet Service Provider (Comcast, Windstream, etc.) is your gateway to the larger network. This is where the "inter" portion of "Internet" comes from - one provider's network doesn't span all possible customers and their networks, so it's necessary for providers to communicate between each other or "interconnect". ISPs have connections to other "enterprise ISPs" that form an Internet backbone. Level 3 is one of these backbone companies. With all these connections a map is needed to tell you how to get from point A to point B, which the Internet has in the form of routing tables and DNS which get shared between ISPs. 

On November 6th, a misconfiguration was introduced by mistake to Level 3's routing information. Before the mistake was located and addressed, the routing information had already been shared by the Autonomous Systems (ASes) to other providers propagating that bad config outside Level 3 to its interconnected ISPs. From there a ripple effect took place as the routing error was picked up by more ISPs. Effectively data was being misdirected or "given bad directions" on where it needed to go causing all the connection errors. The effect we see are internet-powered services like streaming, website browsing, email, and VoIP-based communications systems potentially coming to a halt.
Issues such as this can happen for a number of reasons, such as the routing issue seen, issues with DNS, physical cut connections, and even DDoS (capacity overload). The commonality with each of these examples is data being unable to propagate from source to destination. That might sound terribly precarious, but keep in mind the entire internet itself never really goes down, but certain part of the overall network can be affected. Often times this is seen as a slowly loading website. As the internet continues to mature more protection is added and redundancy/contingency services are offered and adopted.

Here at NocTel we use 10 different ISP connections into 4 separate data centers to make sure your data is always flowing. Durning the affected time, we saw limited customers impacted and the impact was caused by local ISP outages and latency issues.

How Can I Prepare for a Future Glitch?

There are a few ways NocTel customers can prepare for these types of events. One is via a direct fiber connection to NocTel that eliminates the "Internet" itself. The direct connection between disparate networks is referred as "peering" and allows the networks to communicate without data needing to traverse across potentially many intermediate hops. Think of peering as having a friend live next door instead of in a different city - reaching this friend is quicker and more reliable when you don't need to rely on transit to get you from your house to theirs and potentially getting lost along the way. We can offer this option to many customers across the U.S..  

Another very good option would be to run multiple ISPs into your organization to ensure failure or overload of one ISP doesn't result in the organization's Internet reliant operations being impaired or coming to a halt. Due to various constraints this is not an option for everyone. In which case we recommend setting up an emergency routing plan for times when your ISP is experiencing issues or even in the event of natural disaster. An emergency plan can be as simple as enabling forwarding for an extension to send calls to an answering service, a cell phone, or to somewhere off-site that is not experiencing ISP issues. We have a detailed guide on call forwarding on our online knowledge base.

For users with more advanced inbound routing setups involving hunt groups or menus, NocTel recommends creating an emergency routing extension in advance. With an emergency routing extension in place, when an issue occurs it's a simple matter of routing your main numbers to the emergency extension. During a service outage, you can change this using a mobile device or by contacting us at NocTel Support to make the routing adjustment. 

If you are interested in setting up an emergency routing extension, please contact support at or 503.764.4300 to talk about options that would be best for you organization. 

All About Caller ID

What is my current Caller ID set as? Are my extensions using the desired DID for outbound calls? How do I update or make changes to my Caller ID name? These are a few questions that NocTel support gets regarding Caller ID and to help our users better understand Caller ID and how to make changes we have a new article on the NocTel Knowledge base.

Helpful Tips and Info on Caller ID:
  • The Caller ID name is the text displayed with a number for outbound calls.
  • The Caller ID name is assigned to a number and can be viewed on a number’s details page in the NocTel control panel.
  • Caller ID names are limited to 15 characters.
  • Caller ID names must adhere to FCC regulations and CNAM database policies.
  • It can take up to 60 days for changes to a Caller ID name to be updated in the CNAM database and distributed to all carriers on the telephone network.
  • Once the Caller ID name is set to a number, the telephone number can be associated with an extension for its outbound caller ID.
Please check out the Caller ID article for more details and guides to making changes to your account's Caller ID. If you need further assistance the support team is standing by to help. Contact us at
Copyright © 2017 NocTel Communications, Inc., All rights reserved.

Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp