Medium - Network Maintenance Notification (Resolved)

Affecting Other - Datacenter AVL01 Network upgrades | Priority - Medium

Maintenance Scheduled for AVL01 (Asheville)
Maintenance Time:  01:00 AM to 02:00 AM Eastern Time (05:00 to 06:00 UTC/GMT) Sunday April 24, 2011
Maintenance Window Duration: 1 hour
Maintenance Action:  Network upgrades
Site(s) Affected:  AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 573995

Netriplex network operations will be performing maintenance on network infrastructure in Asheville (AVL01) this coming Sunday, April 24, 2011. During this window, engineers will install an additional 48-port gigabit module in order to increase capacity to support future network growth. Additionally, they will be power-cycling another 48-port gigabit module that is presently in a minor error state.  For approximatley 3 minutes during this maintenance window, customers may experience momentary periods of packet loss or latency. We will perform this work between 01:00 AM and 02:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

 

Thank you,

Cyberdog Network Team

 

(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Date - 04/23/2011 10:00 - 04/24/2011 11:00

Last Updated - 11/12/2011 06:36

Low -Network Maintenance: ATL01, AVL01 (Resolved)

Affecting System - ATL01, AVL01 Datacenters | Priority - Low

Maintenance Scheduled for ATL01 (Atlanta)
Maintenance Time:  01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Sunday January 2, 2011
Maintenance Window Duration: 4 hours
Maintenance Action:  Network upgrades
Site(s) Affected:  ATL01, AVL01
Risk Classification: Low
Cyberdog Master Ticket #: 557836

Netriplex network operations will be upgrading the IOS on our core routers in Atlanta (ATL01) this coming Sunday, January 2, 2011. During this window, engineers will fail over AVL01 traffic from Atlanta to Ashburn to avoid Customer impact, however customers may experience brief periods of packet loss or latency as engineers perform these momentary failovers. We will perform this work between 01:00 AM and 05:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

 

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Date - 01/02/2011 01:00 - 01/02/2011 05:00

Last Updated - 01/13/2011 16:05

Cyberdog Memeber Center Miantenance (Resolved)

Affecting Other - secure.rakserve.com | Priority - Low

The Cyberdog Member Center (secure.rakserve.com) will be going down for maintenance on Sunday, January 15th, 2010 between midnight and 1am EST.  During this time you may not be able to access the Member Center.  Maintenenace is expected to go quickly but an hour has been scheduled for this upgrade.  IF you have any questions or concerns, please contact our support department.  Thank you!

Date - 11/01/2020 16:00 - 04/21/2011 15:21

Last Updated - 01/13/2011 16:05

Network Outage for AVL01 Facility (Resolved)

Affecting System - AVL01 Datacenter | Priority - Critical

15:51 EST- A network outage has been reported.

16:10 EST- Call placed to datacenter, awaiting response.

16:25 EST- A Major trunk link to ATL01 is down, network engineers are currently attempting to re-route traffic via updating their routing tables. No ETA given at this time.

16:36 EST- Outage has exceeded three and four "9s" SLAs.  To have your account credited accordingly please open a support ticket with the Billing Department.  NOTE: All servers are stable and have been up for 4+ years without one problem or forced reboot, unfortunately the network outage masks this.

17:15 EST- AVL01 is having BGP Session issues, your traffic may still not be routing properly.  All BGP Sessions and peering sessions are down to both Netriplex facilities.

17:30 EST- ETA given as 30 minutes.

18:04 EST- Service restored!

18:06 EST- A Temporary fix has been issued to aid the BGP routing issue (static routes), BGP resolution is still underway...

18:47 EST- Issue fully mitigated. Report to follow.

18:59 EST- Notice from DC: As you are aware, we experienced a major network issue this afternoon that started at around 15:45 Eastern with a few BGP flaps followed by a full-on outage starting at 15:55.  While service has been restored, we're not even close to closing out this issue. I am meeting with our network infrastructure team members right now to fully understand what transpired so I can provide you with the full details. We have a case open with Cisco TAC as well to bring additional insight into the events that transpired to cause an outage of this magnitude. Please look for an update from me within the next few hours.

Dec 16 2010 0:47 EST- Notice from DC (snippet): Unfortunately, the true cause is yet to be determined as of this writing. What is certain, is that since BGP is not currently implemented between our edge and core at tis time, the issue cannot repeat itself right now, which is somewhat of a relief.  Only once Cisco and our engineers have determined what happened and how to ensure it can never happen again, will we consider implementing BGP between these layers again. Both Cisco TAC and Netriplex engineers believe that a Cisco IOS bug is to blame, but we have yet to confirm that this is definitely the case.

Many customers have written me personally or have posted comments in public forums questioning our redundancy, and for good reason. However, rest assured that AVL01 has no shortage of router and switching redundancy. If you’ve seen our network diagram, you know our infrastructure is exceedingly robust. Even if we doubled our infrastructure, the issue would have still occurred since a dropped route announcement would have instantaneously propagated throughout any number of routers. The bigger challenge today is to determine what we can do to prevent this exact scenario from happening again. Once we have accomplished that, we can expand our discussion in an attempt to determine if something slightly different could affect our network in a similarly catastrophic way.

Dec 17 2010 11:04 EST- Notice from DC (snippet): Working with Cisco TAC, our engineers believe we have narrowed down the cause of Wednesday’s outage to a known bug affecting the Cisco 6708 module when deployed in 7600 series routers. The 6708 module has 8 x 10 Gigabit ports that we use to cross uplink our edge to our core. All of the 10G connections on one particular module began to cycle up/down sporadically 10 minutes prior to the outage. While this up/down activity did not specifically cause any connectivity issues due to the redundancies in place, we believe that it was the ultimate trigger to the BGP failure between our edge and core which did cause the outage.

Emergency Maintenence Scheduled for Sunday December 19 beginning at 1:AM EST. View Here

Date - 12/15/2010 15:51 - 12/15/2010 18:04

Last Updated - 12/23/2010 16:16

Medium -Network Maintenance: AVL01 (Resolved)

Affecting Server - Pug | Priority - Medium

Maintenance Scheduled for AVL01
Maintenance Time:  01:00 AM to 05:00 AM Eastern Time (06:00 to 10:00 UTC/GMT) Monday December 27, 2010
Maintenance Window Duration: 4 hours
Maintenance Action:  Network upgrades
Site Affected:  AVL01
Risk Classification: Low
Cyberdog Master Ticket #: 556761

Netriplex network operations will be implementing additional module uplink diversity between our edge and core infrastructure this coming Monday, December 27th in our AVL01 datacenter. During this window, engineers will fail over traffic to avoid Customer impact, however customers may experience brief periods of packet loss or latency as engineers perform these momentary failovers. We will perform this work between 01:00 AM and 05:00 AM Eastern.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

Thank you,

Cyberdog Network Team

 

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Date - 12/27/2010 01:00 - 12/31/2010 01:57

Last Updated - 12/23/2010 16:15

Medium - Network Maintenance Notification (Resolved)

Affecting Other - Datacenter AVL01 IOS upgrade and 10G module installation | Priority - Medium

Maintenance Scheduled for AVL01
Maintenance Time: 02:00 AM (Eastern, GMT-4) May 23, 2010
Maintenance Window Duration: 15 minutes
Maintenance Action: IOS upgrade and 10G module installation
Site Affected: AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 518138

Netriplex network operations will be performing an IOS upgrade to the core network infrastructure in AVL01 to a more stable release and to facilitate the installation of an additional module to support eight additional 10Gig ports. Customers will experience packet loss, increased latency, or other connectivity issues when traffic shifts to and from redundant connectivity. We will perform this work at shortly after 02:00 AM Eastern time on Sunday, May 23, 2010 and expect it to complete within 15 minutes.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

 

Thank you,

Cyberdog Network Team

 

(Low) The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High) A service disruption should be anticipated. The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Date - 05/23/2010 02:00 - 05/30/2010 02:29

Last Updated - 12/17/2010 18:04

Network Outage for AVL01 Facility (Resolved)

Affecting Server - Pug | Priority - Critical

Status: Reported/Investigating

As of 21:16 PDT there is a network outage effecting our AVL01 facility servers.  This is a routing problem due to a main trunk line outage.  You may experience in and out connectivity problems.  We will update this ticket when we have more information.

Cyberdog Networking Team

 

Status: Determined

As of 22:39 PDT the cause of the network issue has been determined.  Multiple fiber lines have been accidentally severed during a 3rd party vendor upgrade to a pipe outside of the AVL01 facility properpy line (half mile away).  The vendor is working to fix the lines now.

 

As of 22:45 PDT service has been restored.

Date - 10/04/2010 21:16 - 10/04/2010 22:45

Last Updated - 12/17/2010 00:40

Network Outage for AVL01 Facility (Resolved)

Affecting Server - Pug | Priority - Critical

As of 02:02 PDT there is an unknown network outage effecting our AVL01 facility servers.  This is a routing problem so you may experience in and out connectivity problems.  We will update this ticket when we have more information.

Cyebrdog Networking Team

 

As of 02:32 EDT all systems are nominal.  A report will be logged shortly.

Date - 06/28/2010 02:02 - 07/09/2010 16:03

Last Updated - 10/04/2010 00:44

Medium - Emergency Maintenance Notification (Resolved)

Affecting Other - Datacenter AVL01 Transformer Replacement | Priority - Medium

Maintenance Scheduled for AVL01
Maintenance Time:  07:00 AM (Eastern, GMT-4) May 2, 2010
Maintenance Window Duration: 4 to 6 hours hours
Maintenance Action:  Progress Energy Transformer Replacement
Site Affected:  AVL01
Risk Classification: Moderate
Cyberdog Master Ticket #: 514649

Progress Energy has informed us that they will need to replace a utility transformer behind AVL01 feeding our facility. No downtime is anticipated and all precautions have been implemented. We will switch to stable generator power prior to and for the duration of their maintenance activities. They will perform this activity tomorrow, Sunday May 2, 2010 starting at 7:00 AM and should complete at or around 12:00 Noon.

If you have any questions, or would like to determine the extent of impact any maintenance may have on your network, please contact us.

 

Thank you,

Cyberdog Network Team

 

(Low)  The probability of a disruption in service is minimal. The maintenance being performed involves basic infrastructure systems and should not affect normal operations. Although the risk of a service interruption is minimal, Cyberdog requires this work be performed during the maintenance window as a matter of policy.

(Moderate) There is a possibility of a disruption in service. The maintenance being performed involves mission critical infrastructure systems. While redundant systems are the hallmark of our data center, we wanted to advise you in advance in the event the maintenance does not go as planned. We have taken all prudent precautions to eliminate the likelihood of a service interruption.

(High)  A service disruption should be anticipated.  The maintenance being performed affects mission critical infrastructure systems. Should a disruption in service occur, Cyberdog will move rapidly to limit this interruption to the absolute minimum amount of time. This classification is required for maintenance where the risk of a service outage is anticipated at being greater than 80%.

Date - 05/02/2010 07:00 - 05/02/2010 15:32

Last Updated - 05/20/2010 16:20

Below is a real-time overview of our servers where you can check if there's any known issues.

View RSS Feed