NETWORK RELATED:
ServerOrigin has received a few reports today of intermittent connectivity issues. We have verified the issue and found the problem to be an upstream routing issue. The problem has been identified and we are working to have it corrected very shortly. We want to thank everyone for the notifications and apologize for any inconvenience this caused. It appears this was only affecting specific routes in and out of our facility for short periods of time so it was very difficult to nail down the exact problem. However, now that it has been identified – we should have a full resolution shortly.
We thank you again for your patience and wish everyone a safe and happy holiday season!

No customer data is at risk and there has been no breach of data or services. This is all due to specific DDoS attacks targeting critical network infrastructure which is either causing some intermittent routing or general degradation such as packet loss. The issue has been ongoing for nearly 48-hours now but the problem has now been identified and is being resolved. ServerOrigin currently has our entire support staff and all engineers working on this issue so ticket responses may see a short delay as we are working this as the absolute highest priority.

We apologize for any inconvenience this may cause as we know this time of year is particularly important for all of our customers. We are working feverishly to get the issue resolved and to ensure the least amount of interruption occurs while the changes are being made. We appreciate your patience and understanding. We will up-hold any current Service Level Agreements and ensure all customers impacted receive credit for any downtime during this emergency change period.

The issue is impacting all ethProxy customers specific to the West Coast facility.

ServerOrigin engineering is working diligently to have this resolved and provide an update shortly.

–Update–

21:43 CDT: Services recovered: Working to complete check-out process and will provide a complete cause of outage shortly.

During the above referenced period, datacenter engineers will be updating code on production edge routers. No extended outages are expected, however, the process of re-routing traffic between edge routers in order to take them offline for maintenance is expected to cause very brief outages not expected to exceed several minutes.

This maintenance is necessary to update the JunOS code on the routers in order to correct a bug that has caused issues to the production network on 1 previous occasions in the past month.

11:26PM CDT: Peer1 is having some form of network-related issue as we have are currently taking impact on our Peer1 links. We have the NOC at Peer1 looking into this issue at this time.

All ServerOrigin equipment is functioning properly, we are not receiving any data from Peer1. We will update here shortly.

11:31PM CDT All customers in this location were automatically re-routed to a different facility. All sites come up.

12:40AM CDT All customers moved back after link was repaired.

12:51AM CDT Final Checkout completed, customers experienced approximately 8 minutes of downtime.

You can still reach support via: https://billing.serverorigin.com
Email: support@serverorigin.com
Phone: 1-888-670-0853

**No customers are impacted with this downtime. This is simply a brief down due to relocation of our company website.

Thanks so much!
ServerOrigin Communications

Abstract: ServerOrigin sustained a brief network outage resulting from an unforeseen error in the DDoS filtering array which caused connectivity issues for many users.

Cause: A scheduled upgrade for the expansion of the filtering array including additional bandwidth links from Peer1. During installation traffic was moved from the primary array to the secondary. Once moved all filtering continued but on reduced bandwidth. ServerOrigin encountered multiple DDoS attacks saturating the backup array and causing intermittent connectivity and dropped packets.

Resolution: Traffic was re-routed through the primary and secondary filtering arrays with the changes saved for completion tomorrow during non-peak hours. No further downtime or intermittent connectivity is expected.

We apologize for the brief connectivity issues seen lately and we want to assure you that we are aware and working to resolve it. The end of 2010 and the beginning of 2011 has brought larger attacks with higher PPS rates which appear to be able to sustain for longer periods of time than averages seen prior to last quarter ’10. Due to the volume and demand for protection ServerOrigin continues to upgrade and add additional infrastructure but it’s a moving target and one that requires live upgrades. Even with thorough testing and check-out, there is always possibility for something not going according to plan and as much as we try to avoid that, we have seen several cases lately where things simply did not go according to plan.

We thank you as a ServerOrigin customer for your patience and support. This upgrade will bring an additional 100% of overall bandwidth and filtering capacity to handle recent and future growth. Once completed, many of the issues seen over the past few weeks should be fully resolved. Scheduled completion is by Thursday night. We will notify you if any connectivity issues are planned. No downtime is planned for the upcoming maintenance.

–
ServerOrigin Communications
Network Operations
support@serverorigin.com

https://billing.serverorigin.com

ethProxy Subscribers: ethProxy customers are provided fully automated DNS failover and redundancy to one of our other facilities. *IF* you allow ServerOrigin to manage your DNS via our AnyCast DNS service then you will be automatically re-routed to an alternate facility. We will also move customers proactively prior to the changes. If you are not currently taking advantage of this free service,  please notify us or make the appropriate changes so we can handle the DNS failover.

Abstract:  ServerOrigin currently maintains critical infrastructure in two suites at 600 W. 7th St, Los Angeles, CA: Suite 420 (Peer1) and Suite 130 (TelX). Over the course of one week critical infrastructure and some equipment leased to customers will be consolidated into Suite 130. There will be downtime associated with this maintenance; the exact amount will vary from customer to customer with a total impact of 5 – 30 minutes per customer for the entire week.

This is a preliminary notice and there may be changes to the following timeline as we move closer to the maintenance window:

Chronology:

05 DEC 2010, 3PM – 10PM, U.S. Pacific: Many cloud hosted services such as DDoS protection proxies and all Cloud VM instances will be offline during a physical equipment move within this window not expected to exceed 30 minutes. Any such instances should not experience further substantial issues outside of this window.

06 DEC 2010, 12PM – 4PM, U.S. Pacific: There will be edge router maintenance during this period not expected to result in service impact.

07 DEC 2010, 8AM – 10PM U.S. Pacific: There will be a substantial amount of routing changes and physical relocation of edge routers and DDoS mitigation appliances. Service impact is not expected, however, this change is considered high risk and unexpected outages could occur. Our Telia transit will be offline during currently undetermined periods due to local loop maintenance.

08 DEC 2010, 7AM – 5PM: Many customer leased servers will be relocated, resulting in a service impact not expected to exceed 30 minutes per server.

09 DEC 2010, 7AM – 5PM: Many customer leased servers will be relocated, resulting in a service impact not expected to exceed 30 minutes per server.

10 DEC 2010, 8AM – 10PM U.S. Pacific: This window is reserved for final routing changes, network testing, and quality assurance. Service impact is not expected at this time.

11 DEC 2010, 8AM – 10PM U.S. Pacific: This window is reserved for quality assurance and last minute changes. Service impact is not expected at this time.

We apologize for the inconvenience as this matter is being resolved.

**NO CUSTOMERS ARE IMPACTED DUE TO THIS ISSUE. THIS IS ONLY SERVERORIGIN.COM ITSELF**

Thanks for your patience.

Please email support@serverorigin.com if you have any questions!

Maintenance will end by 9AM CDT today.