LIME Barbados - Some Corporate Customers Lose Online Services for 2 Days due to DNS Server Issues

Date: 2012-Feb-13

Overview

For a couple of days, Feb 6th to mid-morning Feb 8th, 2012, certain corporate customers of LIME (Cable & Wireless) in Barbados may have been affected by a DNS (Domain Naming System) server issue at LIME.

It likely caused the loss of two and a half (2 and 1/2) days of Internet productivity (or entertainment) for the affected corporations and constituent users.

Those experiencing the problem would have endured a complete outage of any Internet service that required DNS to function! This includes any web pages accessed using a web browser.

Outage Details

Apparently, the LIME DNS servers 205.214.192.201 and 205.214.192.202 were not answering DNS queries from the LIME Broadband network. Thus, LIME's customers could not directly query them.

This was not an Internet outage. It was a DNS outage. The problem was correctable, once known, both at end-user machines or by the Systems Administrator reconfiguring the server and appropriate network equipment. It only affected companies with a specific collection of configuration settings. It is assumed the number of affected companies was large due to the common nature of the configuration.

It would take about three (3) minutes for an end-user to be guided into working around the issue once permitted to modify the network adapter settings.


Typical Static DNS Servers Configured Systems
Traditionally, the above DNS servers were statically configured in:

• The Internet Protocol (IP) properties on on the network adapter of Windows Server machines functioning as DNS servers, Internet Security & Acceleration (ISA) servers, routers.
  


• Hardware routers.
  


• Hardware firewalls.
  


• Microsoft DNS server settings as the Forwarders.
  


• Distributed to Windows client computers via the DHCP's DNS options.

Concerns Emanating from the Outage

This outage should concern all who were affected in the following ways:

1. It lasted in excess of two (2) days! Estimated outage time is a minimum of 53 hours.



2. LIME's DNS servers col1.caribsurf.com [205.214.192.201] and col2.caribsurf.com [205.214.192.202] are authoritative name servers for serveral local domains (the correct term to use here is DNS zones). Whilst those servers were resolving DNS queries of international origin (according to LIME and confirmed by testing), they failed to resolve DNS queries from LIME's customers. This meant that if the customers' local caching DNS server was either, 1.) using root hints to resolve queries or 2.) using those afflicted LIME name servers as Forwarders, it would fail to communicate with any of the LIME hosted DNS zones. However, DNS zones not hosted by LIME would have been accessible with the root hints configuration.



3. Whilst LIME has provisioned another caching DNS server dns.caribsurf.com [205.214.222.201], it is not an authoritative name server. That is to say it is not secondary to the primary name server for the hosted DNS zones.



4. Both of the LIME Barbados' authoritative name servers (primary and secondary), as known to the public, appear to be on the same IP subnet. This makes the network path (or some part of it) a single point of failure, and thus minimizes the engineered redundancy. A failure on the upstream router to these name servers, and as was likely Feb 6th - 8th, 2012 a misconfiguration of the network policy on the upstream router can cause an outage. An outage could affect all DNS operations (authoritative and caching DNS service to the Internet) or authoritative and caching DNS operations only on the LIME network.
   


5. On the issue of public relations by LIME during this critical network outage event, this was as absymal as usual. LIME failed to list this issue as a Service Alert [1]. However, the single day closure of their retail outlets on Feb 10th, 2012 due to a Union meeting was important to list (see image at top of page).

REFERENCES

[1] Service Alerts- LIME Barbados. http://www.time4lime.com/bb/news/service_alerts.jsp

2009-09-12 – LIME St. Lucia – Blocks SMTP Communication – Outbound Traffic on Port 25 – Disrupts a Business from Sending E-mail for at Least 1 Week!

What Has LIME St. Lucia Done?

LIME St. Lucia has blocked the e-mail communication in a manner that stops Asynchronous Digital Subscriber Line (ADSL) Internet service subscribers from being able to send e-mail from desktop (or end-user) e-mail clients such as Microsoft Outlook, Microsoft Outlook Express, Mozilla Thunderbird, Eudora etc.

An Introduction to ESMTP

Extended Simple Mail Transfer Protocol (ESMTP) is a protocol used to transport Internet mail. It is used both as:

1. An inter-server transport protocol (transfer of messages between mail servers on the Internet) and,

2. As a mail submission protocol (transfer of messages from end-user e-mail clients to their subscribing mail server - often with restricted behaviour enforced).

The protocol operates on Transmission Control Protocol (TCP) port 25 [http://en.wikipedia.org/w/index.php?title=Extended_SMTP&oldid=312435768].

Message Submission and Secure SMTP

For years, port 25 has been the well known port for Simple Mail Transfer Protocol (SMTP) communication. However, under the weight of Internet spam, concerted efforts were made to separate inter-server transport from mail submission via the development of a message submission protocol (see RFC 4409 - Message Submission for Mail (April 2006) at http://tools.ietf.org/html/rfc4409) and the use of a separate well known port for this function. In essence, very similar software, and essentially the same SMTP protocol, is used for both functions. The separation of the two (2) functions lends to better e-mail management and security policies.

The adoption of a Message Submission specific port, notably port 587, as well as other other secure SMTP communication ports such as for SMTP over SSL or TLS has experienced a quite slow roll-out amongst both E-mail Service Providers (ESP) and end-users. Some system / network / e-mail / security administrators are blissfully unaware of their existence. Clear evidence of this is:

1. The absence of these services on some nationally and internationally popular mail servers and,

2. Firewall policies at some sites that specifically prohibit communication on the ports that these services use.

Thus, port 25 is by usually the most popular port used by E-mail Service Providers (ESP) and by local mail server implementations to communicate both with end-user e-mail software and with other mail servers when sending e-mail.

Concerns with the Blocking of TCP Port 25

It is in this context we address the action by LIME St. Lucia. The major concerns with its action are:

1. The business communication disruption to persons using end-user e-mail client software to communicate with an external, Internet-based mail server to which they are subscribers. Typically, that external mail server performs an e-mail relaying service on behalf of the e-mail subscriber. Most small and medium-size businesses (SMB) with simple local network architectures would have such a set-up, some relying on LIME St. Lucia to perform this service, others relying on some third party provider.

2. The difficulty in troubleshooting this type of issue without explicit knowledge of LIME St. Lucia's change in policy. Otherwise, it could take significant time to stumble on this issue or to conclusively rule out other possibilities.

3. The absence of information pertaining to this policy in the Service Alerts or other informational sections of LIME St. Lucia's web-site (http://www.time4lime.com).

Specific to two (2) known cases of service disruptions emanating from this action by LIME:

1. A business endured a disruption in service for at least 1 week without knowing that the fault lay with its Internet Service Provider (ISP).

2. In another, site Information Technology (IT) personnel initially attributed the fault to a configuration or failure condition in the network-level firewall and, at the very least, wasted time trying to swap devices.

An important question is, how much money do those outage translate into? How many more case of this exist?

Assumably, Internet subscribers of business-class services, as well as those paying for static IP addresses were immune to this network configuration issue. Or alternatively, such persons were suitably and comprehensively informed, inclusive of mitigation measures such as:

1. The provision of a smart host to use to circumvent this SMTP blockage and,

2. Informing on the use of the message submission protocol.

As the author is not based in St. Lucia, further assessment on this is not possible without persons sharing their experience. However, information reaching me suggests this issue also occurred with Internet leased circuit subscribers.

Conclusions

There are several problems with the application of the solution of blocking TCP port 25 in response to whatever network performance of security issues LIME St. Lucia had encountered:

1. There are better solutions for blocking communication from illegitimate mail servers.

• At the destination mail server level employing:

1. DNS Blackhole Lists (DNSBLs) that list dynamically assigned IP addresses can stop unauthenticated SMTP communication attempts from zombie computer systems before they are able to transfer bulky or malicious data to the destination mail server.

2. Grey-listing can slow down or avoid spam sources from being able to successfully transfer messages to a destination mail server, depending on the configuration of the grey-listing and the spam source.

3. Certain e-mail validation / authentication schemes such as Sender Policy Framework (SPF) can be used to reject mail from unauthorized originating mail servers.

• Perhaps LIME's network engineers can identify the users / spam sources that led to this SMTP blocking decision and inform them of their non-compliance with the Acceptable Use Policy (AUP) associated with their service:

1. The captive portal solutions used by LIME Barbados to notify of ADSL modem upgrades may be employed to notify offending users of their situation, or alternatively a simple telephone call or letter.

2. This may be an opportunity to partner with an Information Technology (IT) support organization to offer “for fee” corrective service to subscribers afflicted with malware.

3. Resolution of the issue, or mitigation of it, may involve the use of some host-based or network-level firewall that restricts outbound SMTP traffic, on a per computer or per site basis, to the finite list of valid mail relay servers.

4. Additionally, it would be useful to identify the source computer(s) and application(s) / process(es) performing the malicious SMTP activity and to disinfect the machine(s) whilst enacting measures to avoid a repeat of similar infections.

5. The level of corrective service could be based on client desire and budget.

6. However, a zombie computer may be under-performing for the end-user and making the end-user(s) think that LIME St. Lucia's Internet service is slowly. Therefore, identifying and correcting the real issue could yield reputation benefits.

• Although, I am not specifically sure of operation or availability of the SMTP proxy setting described here, it seems such an operation is likely and could allow network operators to restrict its user base to using SMTP-AUTH communication from its network or otherwise communicate from mail servers with legitimate host names. The network operator would thus funnel SMTP traffic through the proxy and it would reject SMTP traffic once:

1. SMTP-AUTH fails or alternatively,

2. The HELO / EHLO greeting host name does not match the originating IP address when an DNS A record lookup is performed on the host name.

2. It is possible this action of blocking SMTP communication may be considered monopolistic and malicious against third party E-mail Service Providers (ESP), specifically if the Internet Service Provider has blocked SMTP communication to all Message Transfer Agents (MTA) other than its own.

RECOMMENDATIONS

LIME St. Lucia should seriously consider reversing this policy and utilizing other means to handle whatever problem they had. Really, this network policy decision must be informed by the human and the business perspective, especially in terms of productivity loss, cost of outage and cost of remedial IT services.

The use of the message submission TCP port 587 for SMTP-AUTH communication needs to be introduced to the user base and a smart host feature may be provided. However, further concerns may exist with blocking communication to all other mail server providers other than the ISP's own.

A more offender specific - that is targeted blocking, and further corrective action, needs to be employed rather that the user of broad-spectrum and disruptive network policy settings.

If this SMTP blocking is later deemed the only viable long-term solution, there needs to be a notification and education campaign to reduce possible harsh effects to end-user and business place productivity and any attribution of uncompetitive practices to the company. This is especially the case if the consumer does not actually have the real power of choice with respect to any affected business-level Internet service.

LIME St. Lucia should ensure their dynamic IP ranges issued to dial-up and DSL clients are registered with the appropriate DNS-based Blackhole List (DNSBL) e.g. Spamhaus Policy Block List (PBL) http://www.spamhaus.org/pbl/. Such IP addresses should theoretically never be used to operate Mail Transfer Agents (MTA). Most e-mail administrators should expect the previous to be the case.

Any destination mail servers afflicted with a spam problem originating from a LIME St. Lucia IP range should seriously consider improving their e-mail administration and security practices, especially by employing the DNSBL containing a list of dynamic assigned public IP addresses issued by ISPs.

If LIME St. Lucia has an issue with bandwidth utilization for spam activities originating from their subscribers, there is likely another, more suitable, service provider network solution to this problem other than full TCP port 25 blocking.

2009-02-19 - LIME - BARBADOS - Windsor Lodge Area ADSL Service Outage for Users without PPPoE Configuration

Created: 2009-02-21
Last Updated: 2009-02-23

NOTE: The following describes an issue discovered in a specific locality and is not broadly applicable to any other. Please do not adjust your setttings based on this unless timeline, locality, service type and configuration data makes it clearly applicable. PLEASE SEE THE DISCLAIMER.

Discovering LIME's Broadband Network Upgrade - by Trial

On the morning of February 19th, 2009 I discovered that LIME (Landline, Internet, Mobile, Entertainment)), formerly referred to as Cable & Wireless (C&W), had made changes to their Asymmetric Digital Subscriber Line (ADSL) infrastructure in the vicinity of the Windsor Lodge, St. Michael, Barbados area.

My discovery of what actually occurred was made after I was at the site of the second client affected by the service change and had, by the time, partially troubleshoot connection issues with a total of 2 ADSL subscribers and fully resolved an influenced but slightly different issue at 1 other subscriber (issue of a bad password in a new SpeedStream modem). The previous client count is inclusive of the client I was at when I was able to conclusively realise the nature of the service change, and thus move that client from the stage of partial troubleshoot, to full issue resolution.

Interestingly enough, at the site of the first ADSL service problem I engaged the services of LIME's Call Centre. Despite communicating all relevant fault observations to the Call Centre agent, she assured me that the solution was to replace the Starbridge Pyxis 210 ADSL modem with a free new one from LIME's Windsor Lodge Annex (via trade in). Despite questioning on if I could not resolve the issue at the moment on the pre-existing equipment, and some ranting about the impact of the disruption on business productivity, she offered no further helpful technical details. I resigned myself to comply and to later observe the "magic stuff" in the replacement modem so I could make any applicable changes when other specialized ADSL customer premises equipment (CPE) - as some businesses may have - require similar adjustment.


How an ADSL Upgrade Becomes an Outage

Specifically, LIME's service change would have caused a service outage to ADSL subscribers with ADSL modems that were not configured for Point-to-Point Protocol over Ethernet (PPPoE) on Virtual Path Identifier (VPI) 0 (zero) and Virtual Channel Identifier (VCI) 36 (thirty six).

The issue usually would manifest itself as one in which any status indicators on the the ADSL modem would:

1. indicate a successful DSL link but,
   
2. suggest a failure of Internet communication or in Point-to-Point Protocol (PPP) authentication.

Usually both of these conditions, are represented by appropriately labelled light emitting diodes (LED) on the front bezel of ADSL modems.

Apparently the issue would have more likely affected persons with older modems, which would usually not have been configured in the above described manner by Cable & Wireless / LIME. Additionally, persons and businesses using their own ADSL modem and possibly lacking the PPPoE configuration described would also have been affected. Of course, the issue is not limited to LIME's ADSL subscribers, because Sunbeach Communications Inc. uses LIME's ADSL infrastructure to deliver the service to their customer base. Additionally, Sunbeach Communications Inc. was at some point in time in the habit of requiring that customers acquire their own ADSL modems, so they may assumably have the larger base of customer acquired and self-configured ADSL equipment.

Complicating ADSL issues further is the observation that the layout of the web management interface of certain ADSL modems makes it difficult for an unfamiliar user to change the PPP settings on the appropriate Wide-Area Network (WAN) protocol/service (e.g. PPPoA versus PPPoE when both exist). The LIME distributed SIEMENS Gigaset modems' web management user interfaces (UI) requires careful and correct interpretation to avoid innocent misconfiguration. Essentially, the Point-to-Point Protocol (PPP) authentication settings (i.e. username and password) are configured on a per line protocol basis. Any data entry errors in the username or password on the new line protocol (PPPoE) would result in LIME's authentication infrastructure detecting an invalid authentication attempt (e.g. "bad password").

Furthermore, it seems some ADSL modems were preconfigured by LIME with some initial default PPP authentication settings such as a username of "adslpppoe" and possibly similar password. The effect of this was to make the subscriber have to verify his or her account on the first attempt to browse the Worldwide Web (WWW). Subsequently, LIME's infrastructure attempted to auto-configure the device with the appropriate user settings via TR-069 device management protocol. Sometimes, this activity repeatedly fails and thus may prevent the end-user from being able to browse pass the first time account verification prompt or a bad password error page. Some pages on the LIME web server are however still accessible if one wishes to restrict viewing to these.


LIME's Prior Notice

LIME notified customers of the upgrade in the media. For example, page 22A in the February 8th, 2009 edition of the Sunday Sun, indicated that they are upgrading their broadband network and told the user what to do should they see a particular LIME web page come up. That web page allowed users to check if their ADSL modem required changing.


Contemporary Internet Usage

Internet access is currently finely interwoven into the fabric of life of many residential users and certainly important for many businesses. In some cases, unfettered access in the home is just as important as in the office because a growing number of persons would be accessing work related systems from the comfort of their home, sometimes with high and hopeful expectations of availability, and tight work deadlines.


Remember: ADSL Service is Used in the SMB Market

ADSL service due to it relative affordability versus all other "business class" Internet connectivity products is quite popular in the small/medium-sized businesses (SMB) in Barbados. Often ADSL service is cheaper that other business class Internet connectivity services by a factor of 10!

As businesses grow in size in terms of number of employees and computing devices (and thus network nodes) the network in the office may evolve to incorporate firewalls and servers and often the network architecture may place demands on the services that must be offered by the Wide Area Network (WAN) device. In the case of ADSL, the ADSL modem is the WAN device.

The corporate-resident ADSL modem may be expected to have a certain static Internet Protocol (IP) address, function as a Dynamic Host Configuration Protocol (DHCP) server on a certain range of IP addresses, or not provide DHCP services at all, forward incoming Internet traffic on some ports to certain internal network devices, and a myriad of other site specific requirements. The meaning of all the above can be captured in this statement:

"A corporate ADSL modem is often not a commodity device that can be blindly swapped due to the WAN provider upgrading they network, it is usually a device with a specially configured role within the corporate network."

Of course, each SMB should have access to competent network management service/team that allows the company's WAN service to be as highly available as possible in spite of the whims, technical and customer service limitations of any Internet Service Providers (ISP). I like to regard this as a form of insurance - Information and Communication Technology (ICT) insurance by doing business with a competence network service team.


Issues with the Methods of LIME

I think the customer service person I spoke to at LIME could have certainly released more information on the nature of the upgrade. Further, she was clearly misinformed or misinforming me when she stated the Starbridge Pyxis 210 modem had to be changed for the service to be restored - I successfully reconfigured one at the second customer. I certainly hope other customers had a better experience.

The mass media approach of broadcasting this service upgrade may have been useful to some, but I was actually unaware of it prior to observing the issue. I think the method was too impersonal. Also, I think there was an opportunity to provide:

1. information for the average users,
2. more information to the technical user and
3. useful information for business users to stave off protracted, unplanned service outages.

By providing a means for technical users to get further details and sending the notice directly to subscribers the downtime experienced by some could have been avoided or reduced.


What Could LIME Have Done Differently?

What could LIME have done to minimize the service disruption, where applicable?

1. Keep the mass media approach, it probably helps the local economy. It was however an incomplete communique, and an incomplete solution when a piece of properly packaged, technical communication was also required. I regard it as a Corporate Communication or Marketing response, with glaringly inadequate Engineering input. If the electricity provider turns off the power supply during an upgrade, I may have to wait until they turn it back on. However, if my ISP changes a system, and a settings change in a device (pass the service provider's demarcation point thus, on my property) can restore my service (rather than unplugging a device, communing to LIME, queuing, waiting, communing the return trip, plugging up the new device), then why would they not instruct me to do so? Then they can offer me more leisure time to complete the "nice to have" device replacement option.
2. Use the LIME website to show a detailed notice consisting of timelines of the upgrades and areas affected.
3. Utilize the technology that redirects web page requests to that Modem Upgrade page to actually offer more information on: the settings change, which modem models must be replaced immediately, upgrade timelines et cetera and possibly only redirect to the Modem Upgrade page if the ADSL modem resides in the affected upgrade area (it is highly possible the later was the case).
4. The press advertisement on the Modem Upgrade should have published a Uniform Resource Locator (URL) for curious users to self-check if they require a modem upgrade. This should reduce the number of unnecessary Customer Service calls due to anxiety over the ad.
5. When sending customers their bill, insert a notice on the upgrade, with at least a URL (Uniform Resource Locator) with further details on the configuration changes. The details could list per ADSL modem model instructions on how to change the settings to be "upgrade-proof". If particular ADSL modems cannot work with the upgrade, or have limitations, this should also be communicated.
6. Corporate customers should be advised to seek the advice of their ICT service team. This is because customized network infrastructure would normally exists in a business place and an ICT layperson handling any device replacement may be unable to complete the change without causing a service outage and possibly losing undocumented infrastructure specifics that may have resided on the traded-in old ADSL modem. Of course undocumented network details would be the customers' fault.
7. Send e-mail notices to customers.
8. Give customers a thoughtful timeline for engaging the free device replacement service bearing in mind that device swapping or actual replacement may be best done outside of normal work hours and on weekend days. Additionally, home users experiencing the outage after the first day or two after the upgrade may not realise the outage until in the after work hours.
9. Inform ICT service companies about the "service upgrade" so they could assist their customer base.

DISCLAIMER

All information provided in this document is "as is", with no specific warranties, expressed or implied as to applicability or accuracy. The author does not recommend any actions based on this information and therefore does not accept any liability whatsoever based on reader actions.

All trademarks, tradenames and registered trademarks are the property of their respective holders.