LIME Barbados - Some Corporate Customers Lose Online Services for 2 Days due to DNS Server Issues

Date: 2012-Feb-13

Overview

For a couple of days, Feb 6th to mid-morning Feb 8th, 2012, certain corporate customers of LIME (Cable & Wireless) in Barbados may have been affected by a DNS (Domain Naming System) server issue at LIME.

It likely caused the loss of two and a half (2 and 1/2) days of Internet productivity (or entertainment) for the affected corporations and constituent users.

Those experiencing the problem would have endured a complete outage of any Internet service that required DNS to function! This includes any web pages accessed using a web browser.

Outage Details

Apparently, the LIME DNS servers 205.214.192.201 and 205.214.192.202 were not answering DNS queries from the LIME Broadband network. Thus, LIME's customers could not directly query them.

This was not an Internet outage. It was a DNS outage. The problem was correctable, once known, both at end-user machines or by the Systems Administrator reconfiguring the server and appropriate network equipment. It only affected companies with a specific collection of configuration settings. It is assumed the number of affected companies was large due to the common nature of the configuration.

It would take about three (3) minutes for an end-user to be guided into working around the issue once permitted to modify the network adapter settings.


Typical Static DNS Servers Configured Systems
Traditionally, the above DNS servers were statically configured in:

• The Internet Protocol (IP) properties on on the network adapter of Windows Server machines functioning as DNS servers, Internet Security & Acceleration (ISA) servers, routers.
  


• Hardware routers.
  


• Hardware firewalls.
  


• Microsoft DNS server settings as the Forwarders.
  


• Distributed to Windows client computers via the DHCP's DNS options.

Concerns Emanating from the Outage

This outage should concern all who were affected in the following ways:

1. It lasted in excess of two (2) days! Estimated outage time is a minimum of 53 hours.



2. LIME's DNS servers col1.caribsurf.com [205.214.192.201] and col2.caribsurf.com [205.214.192.202] are authoritative name servers for serveral local domains (the correct term to use here is DNS zones). Whilst those servers were resolving DNS queries of international origin (according to LIME and confirmed by testing), they failed to resolve DNS queries from LIME's customers. This meant that if the customers' local caching DNS server was either, 1.) using root hints to resolve queries or 2.) using those afflicted LIME name servers as Forwarders, it would fail to communicate with any of the LIME hosted DNS zones. However, DNS zones not hosted by LIME would have been accessible with the root hints configuration.



3. Whilst LIME has provisioned another caching DNS server dns.caribsurf.com [205.214.222.201], it is not an authoritative name server. That is to say it is not secondary to the primary name server for the hosted DNS zones.



4. Both of the LIME Barbados' authoritative name servers (primary and secondary), as known to the public, appear to be on the same IP subnet. This makes the network path (or some part of it) a single point of failure, and thus minimizes the engineered redundancy. A failure on the upstream router to these name servers, and as was likely Feb 6th - 8th, 2012 a misconfiguration of the network policy on the upstream router can cause an outage. An outage could affect all DNS operations (authoritative and caching DNS service to the Internet) or authoritative and caching DNS operations only on the LIME network.
   


5. On the issue of public relations by LIME during this critical network outage event, this was as absymal as usual. LIME failed to list this issue as a Service Alert [1]. However, the single day closure of their retail outlets on Feb 10th, 2012 due to a Union meeting was important to list (see image at top of page).

REFERENCES

[1] Service Alerts- LIME Barbados. http://www.time4lime.com/bb/news/service_alerts.jsp