Network Status News & Information

AVL01 Is having the customer firewall for servers Pug & Boxer upgraded.  A 2 hour window has been scheduled but downtime is expected to be much less.  Maintence shall begin at 01:00 EST Monday, December 14, 2015.  This upgrade is to improve customer network performance and security.

-Network Operations

Scheduled Maintenece to Occur May 16th.  A message from the datacenter is included below.

71848 Immedion is committed to providing our customers the highest quality, most resilient network connectivity options. To achieve this mission, Immedion has scheduled an upgrade to enhance the performance of our network. During this maintenance window, Immedion will be upgrading 2 Distribution Routers. As a result, our customers could experience multiple periods of downtime, none greater than 15 minutes between the hours of 12:00 am and 6:00 am Eastern. Immedion will provide periodic updates throughout this process.

8:47 EST: While Immedion engineers have concluded our maintenance activities, we have had reports of a small number of networks not routing properly. We are currently investigating our routing table and advertisements to our upstream providers in our AVL data center.

01:57 EST - Immedion Network Engineering is aware of a very large routing issue affecting Level(3), their peers, and various other carriers across the US. The routing issue is causing packet loss, route propagation problems, and higher than normal latency for some Immedion customers attempting to reach our datacenters. Immedion has removed Level(3) from our BGP topology to minimize the direct impact felt from this event. We are engaged with Level(3) and will not re-introduce their routing table until they confirm the issues are resolved.

Server Pug is offline after a forced reboot.  cPanel and the datacenter have been contacted to resolve this.

Pug is undergoing an upgrade of MySQL 5.1 to 5.5 to address reported stability issues.

Edge and Core router upgrades.

Maintenance Scheduled for AVL01
Date:  05/13/2012
Ticket #: 629455
Expected Duration:  6 Hours
Timeframe:  12:00AM – 6:00AM
Data Center(s):  Asheville, NC
Event Type:  Scheduled Maintenance
Component(s):   Edge / Core Internet Network
Potential Impact:  Major Impact Anticipated

Immedion Network Engineering will be performing maintenance activities on the Edge and Core routers in the Asheville, NC datacenter.  This maintenance will add capacity and improve stability in the datacenter edge and core network.
Due to the type of maintenance there will be a number of BGP related convergence events which will cause interruptions in customer facing internet connectivity for up to 5 minutes at a time.  There will be a number of these events during this maintenance which will happen at unspecified times.  There will be no impact to private or dedicated circuits coming into customer equipment as a result of these maintenance activities.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Edge and Core router upgrades.

Maintenance Scheduled for AVL01
Date:  05/12/2012
Ticket #: 629454
Expected Duration:  6 Hours
Timeframe:  12:00AM – 6:00AM
Data Center(s):  Asheville, NC
Event Type:  Scheduled Maintenance
Component(s):   Edge / Core Internet Network
Potential Impact:  Major Impact Anticipated

Immedion Network Engineering will be performing maintenance activities on the Edge and Core routers in the Asheville, NC datacenter.  This maintenance will add capacity and improve stability in the datacenter edge and core network.
Due to the type of maintenance there will be a number of BGP related convergence events which will cause interruptions in customer facing internet connectivity for up to 5 minutes at a time.  There will be a number of these events during this maintenance which will happen at unspecified times.  There will be no impact to private or dedicated circuits coming into customer equipment as a result of these maintenance activities.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Emergency Upgrade to IOS and modules to prevent BGP Failure.

Maintenance Scheduled for AVL01
Maintenance Time:  01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Sunday December 19, 2010
Maintenance Window Duration: 4 hours
Maintenance Action:  Network upgrades
Site Affected:  AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 555892

Netriplex network operations will be performing IOS upgrades and routing changes to edge and core infrastructure this coming Sunday, December 19th in our AVL01 datacenter. During this window, engineers will reroute and fail over traffic to minimize Customer impact, however customers may experience periods of packet loss, latency or very brief service interruptions as engineers perform these upgrades. There will NOT be an extended outage during this maintenance window. We will perform this work between 01:00 AM and 05:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

23:43 EST- A network outage has been reported.

23:48 EST- Network is back up, we will continue to monitor for changes.

00:17 EST- A network outage has been reported.

00:19 EST- Call placed to datacenter, no ETA given but aware of the network event.

00:23 EST- Routing issue has been identified, intermittent packet loss may occur.

00:44 EST- Service restored!

06:04 EST- A network outage has been reported.

06:12 EST- Call placed to datacenter, they are aware of the network flapping (up/down) and will report when they have more information from the engineers.

06:27 EST- Service restored!

Maintenance Scheduled for AVL01 (Asheville)
Maintenance Time:  01:00 AM to 02:00 AM Eastern Time (05:00 to 06:00 UTC/GMT) Sunday April 24, 2011
Maintenance Window Duration: 1 hour
Maintenance Action:  Network upgrades
Site(s) Affected:  AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 573995

Netriplex network operations will be performing maintenance on network infrastructure in Asheville (AVL01) this coming Sunday, April 24, 2011. During this window, engineers will install an additional 48-port gigabit module in order to increase capacity to support future network growth. Additionally, they will be power-cycling another 48-port gigabit module that is presently in a minor error state.  For approximatley 3 minutes during this maintenance window, customers may experience momentary periods of packet loss or latency. We will perform this work between 01:00 AM and 02:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Maintenance Scheduled for ATL01 (Atlanta)
Maintenance Time:  01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Sunday January 2, 2011
Maintenance Window Duration: 4 hours
Maintenance Action:  Network upgrades
Site(s) Affected:  ATL01, AVL01
Risk Classification: Low
Cyberdog Master Ticket #: 557836

Netriplex network operations will be upgrading the IOS on our core routers in Atlanta (ATL01) this coming Sunday, January 2, 2011. During this window, engineers will fail over AVL01 traffic from Atlanta to Ashburn to avoid Customer impact, however customers may experience brief periods of packet loss or latency as engineers perform these momentary failovers. We will perform this work between 01:00 AM and 05:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

The Cyberdog Member Center (secure.rakserve.com) will be going down for maintenance on Sunday, January 15th, 2010 between midnight and 1am EST.  During this time you may not be able to access the Member Center.  Maintenenace is expected to go quickly but an hour has been scheduled for this upgrade.  IF you have any questions or concerns, please contact our support department.  Thank you!

15:51 EST- A network outage has been reported.

16:10 EST- Call placed to datacenter, awaiting response.

16:25 EST- A Major trunk link to ATL01 is down, network engineers are currently attempting to re-route traffic via updating their routing tables. No ETA given at this time.

16:36 EST- Outage has exceeded three and four "9s" SLAs.  To have your account credited accordingly please open a support ticket with the Billing Department.  NOTE: All servers are stable and have been up for 4+ years without one problem or forced reboot, unfortunately the network outage masks this.

17:15 EST- AVL01 is having BGP Session issues, your traffic may still not be routing properly.  All BGP Sessions and peering sessions are down to both Netriplex facilities.

17:30 EST- ETA given as 30 minutes.

18:04 EST- Service restored!

18:06 EST- A Temporary fix has been issued to aid the BGP routing issue (static routes), BGP resolution is still underway...

18:47 EST- Issue fully mitigated. Report to follow.

18:59 EST- Notice from DC: As you are aware, we experienced a major network issue this afternoon that started at around 15:45 Eastern with a few BGP flaps followed by a full-on outage starting at 15:55.  While service has been restored, we're not even close to closing out this issue. I am meeting with our network infrastructure team members right now to fully understand what transpired so I can provide you with the full details. We have a case open with Cisco TAC as well to bring additional insight into the events that transpired to cause an outage of this magnitude. Please look for an update from me within the next few hours.

Dec 16 2010 0:47 EST- Notice from DC (snippet): Unfortunately, the true cause is yet to be determined as of this writing. What is certain, is that since BGP is not currently implemented between our edge and core at tis time, the issue cannot repeat itself right now, which is somewhat of a relief.  Only once Cisco and our engineers have determined what happened and how to ensure it can never happen again, will we consider implementing BGP between these layers again. Both Cisco TAC and Netriplex engineers believe that a Cisco IOS bug is to blame, but we have yet to confirm that this is definitely the case.

Many customers have written me personally or have posted comments in public forums questioning our redundancy, and for good reason. However, rest assured that AVL01 has no shortage of router and switching redundancy. If you’ve seen our network diagram, you know our infrastructure is exceedingly robust. Even if we doubled our infrastructure, the issue would have still occurred since a dropped route announcement would have instantaneously propagated throughout any number of routers. The bigger challenge today is to determine what we can do to prevent this exact scenario from happening again. Once we have accomplished that, we can expand our discussion in an attempt to determine if something slightly different could affect our network in a similarly catastrophic way.

Dec 17 2010 11:04 EST- Notice from DC (snippet): Working with Cisco TAC, our engineers believe we have narrowed down the cause of Wednesday’s outage to a known bug affecting the Cisco 6708 module when deployed in 7600 series routers. The 6708 module has 8 x 10 Gigabit ports that we use to cross uplink our edge to our core. All of the 10G connections on one particular module began to cycle up/down sporadically 10 minutes prior to the outage. While this up/down activity did not specifically cause any connectivity issues due to the redundancies in place, we believe that it was the ultimate trigger to the BGP failure between our edge and core which did cause the outage.

Emergency Maintenence Scheduled for Sunday December 19 beginning at 1:AM EST. View Here

Maintenance Scheduled for AVL01
Maintenance Time:  01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Monday December 27, 2010
Maintenance Window Duration: 4 hours
Maintenance Action:  Network upgrades
Site Affected:  AVL01
Risk Classification: Low
Cyberdog Master Ticket #: 556761

Netriplex network operations will be implementing additional module uplink diversity between our edge and core infrastructure this coming Monday, December 27th in our AVL01 datacenter. During this window, engineers will fail over traffic to avoid Customer impact, however customers may experience brief periods of packet loss or latency as engineers perform these momentary failovers. We will perform this work between 01:00 AM and 05:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Maintenance Scheduled for AVL01
Maintenance Time: 02:00 AM (Eastern, GMT-4) May 23, 2010
Maintenance Window Duration: 15 minutes
Maintenance Action: IOS upgrade and 10G module installation
Site Affected: AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 518138

Netriplex network operations will be performing an IOS upgrade to the core network infrastructure in AVL01 to a more stable release and to facilitate the installation of an additional module to support eight additional 10Gig ports. Customers will experience packet loss, increased latency, or other connectivity issues when traffic shifts to and from redundant connectivity. We will perform this work at shortly after 02:00 AM Eastern time on Sunday, May 23, 2010 and expect it to complete within 15 minutes.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Status: Reported/Investigating

As of 21:16 PDT there is a network outage effecting our AVL01 facility servers.  This is a routing problem due to a main trunk line outage.  You may experience in and out connectivity problems.  We will update this ticket when we have more information.

Cyberdog Networking Team

Status: Determined

As of 22:39 PDT the cause of the network issue has been determined.  Multiple fiber lines have been accidentally severed during a 3rd party vendor upgrade to a pipe outside of the AVL01 facility properpy line (half mile away).  The vendor is working to fix the lines now.

As of 22:45 PDT service has been restored.

As of 02:02 PDT there is an unknown network outage effecting our AVL01 facility servers.  This is a routing problem so you may experience in and out connectivity problems.  We will update this ticket when we have more information.

Cyebrdog Networking Team

As of 02:32 EDT all systems are nominal.  A report will be logged shortly.

Maintenance Scheduled for AVL01
Maintenance Time:  07:00 AM (Eastern, GMT-4) May 2, 2010
Maintenance Window Duration: 4 to 6 hours hours
Maintenance Action:  Progress Energy Transformer Replacement
Site Affected:  AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 514649

Progress Energy has informed us that they will need to replace a utility transformer behind AVL01 feeding our facility. No downtime is anticipated and all precautions have been implemented. We will switch to stable generator power prior to and for the duration of their maintenance activities. They will perform this activity tomorrow, Sunday May 2, 2010 starting at 7:00 AM and should complete at or around 12:00 Noon.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.