Sunday, February 22, 2009

2009-02-19 - LIME - BARBADOS - Windsor Lodge Area ADSL Service Outage for Users without PPPoE Configuration

Created: 2009-02-21
Last Updated: 2009-02-23

NOTE: The following describes an issue discovered in a specific locality and is not broadly applicable to any other. Please do not adjust your setttings based on this unless timeline, locality, service type and configuration data makes it clearly applicable. PLEASE SEE THE DISCLAIMER.

Discovering LIME's Broadband Network Upgrade - by Trial

On the morning of February 19th, 2009 I discovered that LIME (Landline, Internet, Mobile, Entertainment)), formerly referred to as Cable & Wireless (C&W), had made changes to their Asymmetric Digital Subscriber Line (ADSL) infrastructure in the vicinity of the Windsor Lodge, St. Michael, Barbados area.

My discovery of what actually occurred was made after I was at the site of the second client affected by the service change and had, by the time, partially troubleshoot connection issues with a total of 2 ADSL subscribers and fully resolved an influenced but slightly different issue at 1 other subscriber (issue of a bad password in a new SpeedStream modem). The previous client count is inclusive of the client I was at when I was able to conclusively realise the nature of the service change, and thus move that client from the stage of partial troubleshoot, to full issue resolution.

Interestingly enough, at the site of the first ADSL service problem I engaged the services of LIME's Call Centre. Despite communicating all relevant fault observations to the Call Centre agent, she assured me that the solution was to replace the Starbridge Pyxis 210 ADSL modem with a free new one from LIME's Windsor Lodge Annex (via trade in). Despite questioning on if I could not resolve the issue at the moment on the pre-existing equipment, and some ranting about the impact of the disruption on business productivity, she offered no further helpful technical details. I resigned myself to comply and to later observe the "magic stuff" in the replacement modem so I could make any applicable changes when other specialized ADSL customer premises equipment (CPE) - as some businesses may have - require similar adjustment.


How an ADSL Upgrade Becomes an Outage

Specifically, LIME's service change would have caused a service outage to ADSL subscribers with ADSL modems that were not configured for Point-to-Point Protocol over Ethernet (PPPoE) on Virtual Path Identifier (VPI) 0 (zero) and Virtual Channel Identifier (VCI) 36 (thirty six).

The issue usually would manifest itself as one in which any status indicators on the the ADSL modem would:

  1. indicate a successful DSL link but,
  2. suggest a failure of Internet communication or in Point-to-Point Protocol (PPP) authentication.
Usually both of these conditions, are represented by appropriately labelled light emitting diodes (LED) on the front bezel of ADSL modems.

Apparently the issue would have more likely affected persons with older modems, which would usually not have been configured in the above described manner by Cable & Wireless / LIME. Additionally, persons and businesses using their own ADSL modem and possibly lacking the PPPoE configuration described would also have been affected. Of course, the issue is not limited to LIME's ADSL subscribers, because Sunbeach Communications Inc. uses LIME's ADSL infrastructure to deliver the service to their customer base. Additionally, Sunbeach Communications Inc. was at some point in time in the habit of requiring that customers acquire their own ADSL modems, so they may assumably have the larger base of customer acquired and self-configured ADSL equipment.

Complicating ADSL issues further is the observation that the layout of the web management interface of certain ADSL modems makes it difficult for an unfamiliar user to change the PPP settings on the appropriate Wide-Area Network (WAN) protocol/service (e.g. PPPoA versus PPPoE when both exist). The LIME distributed SIEMENS Gigaset modems' web management user interfaces (UI) requires careful and correct interpretation to avoid innocent misconfiguration. Essentially, the Point-to-Point Protocol (PPP) authentication settings (i.e. username and password) are configured on a per line protocol basis. Any data entry errors in the username or password on the new line protocol (PPPoE) would result in LIME's authentication infrastructure detecting an invalid authentication attempt (e.g. "bad password").

Furthermore, it seems some ADSL modems were preconfigured by LIME with some initial default PPP authentication settings such as a username of "adslpppoe" and possibly similar password. The effect of this was to make the subscriber have to verify his or her account on the first attempt to browse the Worldwide Web (WWW). Subsequently, LIME's infrastructure attempted to auto-configure the device with the appropriate user settings via TR-069 device management protocol. Sometimes, this activity repeatedly fails and thus may prevent the end-user from being able to browse pass the first time account verification prompt or a bad password error page. Some pages on the LIME web server are however still accessible if one wishes to restrict viewing to these.


LIME's Prior Notice

LIME notified customers of the upgrade in the media. For example, page 22A in the February 8th, 2009 edition of the Sunday Sun, indicated that they are upgrading their broadband network and told the user what to do should they see a particular LIME web page come up. That web page allowed users to check if their ADSL modem required changing.


Contemporary Internet Usage

Internet access is currently finely interwoven into the fabric of life of many residential users and certainly important for many businesses. In some cases, unfettered access in the home is just as important as in the office because a growing number of persons would be accessing work related systems from the comfort of their home, sometimes with high and hopeful expectations of availability, and tight work deadlines.


Remember: ADSL Service is Used in the SMB Market

ADSL service due to it relative affordability versus all other "business class" Internet connectivity products is quite popular in the small/medium-sized businesses (SMB) in Barbados. Often ADSL service is cheaper that other business class Internet connectivity services by a factor of 10!

As businesses grow in size in terms of number of employees and computing devices (and thus network nodes) the network in the office may evolve to incorporate firewalls and servers and often the network architecture may place demands on the services that must be offered by the Wide Area Network (WAN) device. In the case of ADSL, the ADSL modem is the WAN device.

The corporate-resident ADSL modem may be expected to have a certain static Internet Protocol (IP) address, function as a Dynamic Host Configuration Protocol (DHCP) server on a certain range of IP addresses, or not provide DHCP services at all, forward incoming Internet traffic on some ports to certain internal network devices, and a myriad of other site specific requirements. The meaning of all the above can be captured in this statement:
"A corporate ADSL modem is often not a commodity device that can be blindly swapped due to the WAN provider upgrading they network, it is usually a device with a specially configured role within the corporate network."

Of course, each SMB should have access to competent network management service/team that allows the company's WAN service to be as highly available as possible in spite of the whims, technical and customer service limitations of any Internet Service Providers (ISP). I like to regard this as a form of insurance - Information and Communication Technology (ICT) insurance by doing business with a competence network service team.


Issues with the Methods of LIME

I think the customer service person I spoke to at LIME could have certainly released more information on the nature of the upgrade. Further, she was clearly misinformed or misinforming me when she stated the Starbridge Pyxis 210 modem had to be changed for the service to be restored - I successfully reconfigured one at the second customer. I certainly hope other customers had a better experience.

The mass media approach of broadcasting this service upgrade may have been useful to some, but I was actually unaware of it prior to observing the issue. I think the method was too impersonal. Also, I think there was an opportunity to provide:
  1. information for the average users,
  2. more information to the technical user and
  3. useful information for business users to stave off protracted, unplanned service outages.

By providing a means for technical users to get further details and sending the notice directly to subscribers the downtime experienced by some could have been avoided or reduced.


What Could LIME Have Done Differently?

What could LIME have done to minimize the service disruption, where applicable?
  1. Keep the mass media approach, it probably helps the local economy. It was however an incomplete communique, and an incomplete solution when a piece of properly packaged, technical communication was also required. I regard it as a Corporate Communication or Marketing response, with glaringly inadequate Engineering input. If the electricity provider turns off the power supply during an upgrade, I may have to wait until they turn it back on. However, if my ISP changes a system, and a settings change in a device (pass the service provider's demarcation point thus, on my property) can restore my service (rather than unplugging a device, communing to LIME, queuing, waiting, communing the return trip, plugging up the new device), then why would they not instruct me to do so? Then they can offer me more leisure time to complete the "nice to have" device replacement option.
  2. Use the LIME website to show a detailed notice consisting of timelines of the upgrades and areas affected.
  3. Utilize the technology that redirects web page requests to that Modem Upgrade page to actually offer more information on: the settings change, which modem models must be replaced immediately, upgrade timelines et cetera and possibly only redirect to the Modem Upgrade page if the ADSL modem resides in the affected upgrade area (it is highly possible the later was the case).
  4. The press advertisement on the Modem Upgrade should have published a Uniform Resource Locator (URL) for curious users to self-check if they require a modem upgrade. This should reduce the number of unnecessary Customer Service calls due to anxiety over the ad.
  5. When sending customers their bill, insert a notice on the upgrade, with at least a URL (Uniform Resource Locator) with further details on the configuration changes. The details could list per ADSL modem model instructions on how to change the settings to be "upgrade-proof". If particular ADSL modems cannot work with the upgrade, or have limitations, this should also be communicated.
  6. Corporate customers should be advised to seek the advice of their ICT service team. This is because customized network infrastructure would normally exists in a business place and an ICT layperson handling any device replacement may be unable to complete the change without causing a service outage and possibly losing undocumented infrastructure specifics that may have resided on the traded-in old ADSL modem. Of course undocumented network details would be the customers' fault.
  7. Send e-mail notices to customers.
  8. Give customers a thoughtful timeline for engaging the free device replacement service bearing in mind that device swapping or actual replacement may be best done outside of normal work hours and on weekend days. Additionally, home users experiencing the outage after the first day or two after the upgrade may not realise the outage until in the after work hours.
  9. Inform ICT service companies about the "service upgrade" so they could assist their customer base.


DISCLAIMER

All information provided in this document is "as is", with no specific warranties, expressed or implied as to applicability or accuracy. The author does not recommend any actions based on this information and therefore does not accept any liability whatsoever based on reader actions.

All trademarks, tradenames and registered trademarks are the property of their respective holders.