12/22/2011 – 01:05AM CDT

NETWORK RELATED:
ServerOrigin has received a few reports today of intermittent connectivity issues. We have verified the issue and found the problem to be an upstream routing issue. The problem has been identified and we are working to have it corrected very shortly. We want to thank everyone for the notifications and apologize for any inconvenience this caused. It appears this was only affecting specific routes in and out of our facility for short periods of time so it was very difficult to nail down the exact problem. However, now that it has been identified – we should have a full resolution shortly.
We thank you again for your patience and wish everyone a safe and happy holiday season!

ServerOrigin customers may experience some intermittent connectivity this afternoon due to an upstream network issue. Currently we have DDoS attacks/scans hitting infrastructure equipment that has caused us to revise some of our current network routing and security policies. These changes should go unnoticed for the most part however you may see a minute or two loss of connectivity at different periods throughout the afternoon as continuous changes are made to implement additional security measures and resolve possible routing vulnerabilities.

No customer data is at risk and there has been no breach of data or services. This is all due to specific DDoS attacks targeting critical network infrastructure which is either causing some intermittent routing or general degradation such as packet loss. The issue has been ongoing for nearly 48-hours now but the problem has now been identified and is being resolved. ServerOrigin currently has our entire support staff and all engineers working on this issue so ticket responses may see a short delay as we are working this as the absolute highest priority.

We apologize for any inconvenience this may cause as we know this time of year is particularly important for all of our customers. We are working feverishly to get the issue resolved and to ensure the least amount of interruption occurs while the changes are being made. We appreciate your patience and understanding. We will up-hold any current Service Level Agreements and ensure all customers impacted receive credit for any downtime during this emergency change period.

We are aware of this issue as of 21:20 CDT. We are working on this issue now and will have an update shortly.

The issue is impacting all ethProxy customers specific to the West Coast facility.

ServerOrigin engineering is working diligently to have this resolved and provide an update shortly.

–Update–

21:43 CDT: Services recovered: Working to complete check-out process and will provide a complete cause of outage shortly.

START: 08 JUN 2011, 12:00PM U.S. Pacific
END: 08 JUN 2011, 6:00PM U.S. Pacific
Total Service Impact: Minimal

During the above referenced period, datacenter engineers will be updating code on production edge routers. No extended outages are expected, however, the process of re-routing traffic between edge routers in order to take them offline for maintenance is expected to cause very brief outages not expected to exceed several minutes.

This maintenance is necessary to update the JunOS code on the routers in order to correct a bug that has caused issues to the production network on 1 previous occasions in the past month.

ISSUE: 12:08AM CDT 06/07/2011 : ServerOrigin customers in the Los Angeles facility experienced some intermittent packet loss for around 30 minutes starting at approximately 12:08AM CDT. This was due to a spanning-tree issue on a Cisco switch that has been planned for removal. Issue was only affecting partial segments of the network.
- Fully resolved at 12:40AM CDT 06/07/2011

ISSUE: 12:59PM CDT 06/07/2011: Same cabinet appears to be experiencing packet loss on this particular segment affected partially early morning. We are working to resolve this issue.

PLANNED RESOLUTION: 13:23 CDT 06/07/2011: We should have the intermittent packet loss resolved in the next few minutes.

Once the packet loss issue is resolved, the offending switch from the outage early morning will be removed and replaced with it’s new upgraded switch. The upgraded switches are in place but were not replacing the older switches yet as this was a planned maintenance in the future. However, we will go ahead and do this replacement today to ensure the issue is fully resolved.

Less than 2 minutes of interruption is planned which would allow for spanning tree reconvergence. This change will may not be noticed at all but could potentially cause between a 1-5 minute outage.

This will fully resolve all issues that we have been having over the past week with this set of VLAN’s tied to this switch.

We apologize for the inconvenience and we thank you for your patience as we resolve this matter.

ServerOrigin Support
support@serverorigin.com

Thursday, March 24, 2011:

11:26PM CDT: Peer1 is having some form of network-related issue as we have are currently taking impact on our Peer1 links. We have the NOC at Peer1 looking into this issue at this time.

All ServerOrigin equipment is functioning properly, we are not receiving any data from Peer1. We will update here shortly.

11:31PM CDT All customers in this location were automatically re-routed to a different facility. All sites come up.

12:40AM CDT All customers moved back after link was repaired.

12:51AM CDT Final Checkout completed, customers experienced approximately 8 minutes of downtime.

ServerOrigin.Com – Primary website is temporarily offline for scheduled maintenance. This is a move to our facility from a previous colo facility. We apologize for any inconvenience, the site will be up shortly.

You can still reach support via: https://billing.serverorigin.com
Email: support@serverorigin.com
Phone: 1-888-670-0853

**No customers are impacted with this downtime. This is simply a brief down due to relocation of our company website.

Thanks so much!
ServerOrigin Communications

START: 27 JAN 2011, 3:28AM U.S. Central
END: 27 JAN 2011, 4:05AM U.S. Central
Total Service Impact: 37 min (Intermittent)

Abstract: ServerOrigin sustained a brief network outage resulting from an unforeseen error in the DDoS filtering array which caused connectivity issues for many users.

Cause: A scheduled upgrade for the expansion of the filtering array including additional bandwidth links from Peer1. During installation traffic was moved from the primary array to the secondary. Once moved all filtering continued but on reduced bandwidth. ServerOrigin encountered multiple DDoS attacks saturating the backup array and causing intermittent connectivity and dropped packets.

Resolution: Traffic was re-routed through the primary and secondary filtering arrays with the changes saved for completion tomorrow during non-peak hours. No further downtime or intermittent connectivity is expected.

We apologize for the brief connectivity issues seen lately and we want to assure you that we are aware and working to resolve it. The end of 2010 and the beginning of 2011 has brought larger attacks with higher PPS rates which appear to be able to sustain for longer periods of time than averages seen prior to last quarter ’10. Due to the volume and demand for protection ServerOrigin continues to upgrade and add additional infrastructure but it’s a moving target and one that requires live upgrades. Even with thorough testing and check-out, there is always possibility for something not going according to plan and as much as we try to avoid that, we have seen several cases lately where things simply did not go according to plan.

We thank you as a ServerOrigin customer for your patience and support. This upgrade will bring an additional 100% of overall bandwidth and filtering capacity to handle recent and future growth. Once completed, many of the issues seen over the past few weeks should be fully resolved. Scheduled completion is by Thursday night. We will notify you if any connectivity issues are planned. No downtime is planned for the upcoming maintenance.

–
ServerOrigin Communications
Network Operations
support@serverorigin.com

https://billing.serverorigin.com

=======================================================================
SoftLayer Emergency Maintenance – GLOBAL [12/30/2010]
Date: Thursday, December 30, 2010 (12/30/2010)
Start Time: 05:00 PM CST
End Time: 07:00 PM CST
Services affected: Public / Private Network
Location: Global
Duration: 2 hours

==================================================
SoftLayer Engineers will be performing emergency maintenance on BBR01.EQ01.WDC02, BBR02.EQ01.WDC02, and BBR01.WB01.SEA01 in an effort to correct routing issues customers are currently experiencing to various destinations including AT&T residential services. In conjunction with Cisco, Engineers have identified a bug that has resulted in the router entering a degraded performance state. To correct this degraded state, the router must be reloaded.

Customers can expect to see latency / packetloss as routes reconverge across redundant routers on both the public and private networks. While the maintenance window is set for 2 hours, we expect the maintenance to take no longer than 20 minutes during each router reload.

Customers in the Seattle datacenter will lose backend connectivity to all other datacenters (local Seattle private network traffic will not be impacted).
==================================================

If you have any problems after this time frame with regard to connectivity, or if you have any questions regarding the maintenance at any point, please open a ticket in the customer portal or contact our support via phone.

We appreciate your patience during this work and welcome any feedback.

Thank you,
Network Engineering
——————————————————————-

This is likely to interrupt a small percentage of ServerOrigin’s Washington, DC / Seattle traffic. We hope this change will be as quick and uninterrupted as possible.

Thanks,
Kyle A.
ServerOrigin Communications

START:  05 DEC 2010
END:  11 DEC 2010
Total Service Impact: Varies depending on maintenance sub-window, see below. (Expected No more than 30 Minutes per customer for the entire week of maintenance)

ethProxy Subscribers: ethProxy customers are provided fully automated DNS failover and redundancy to one of our other facilities. *IF* you allow ServerOrigin to manage your DNS via our AnyCast DNS service then you will be automatically re-routed to an alternate facility. We will also move customers proactively prior to the changes. If you are not currently taking advantage of this free service,  please notify us or make the appropriate changes so we can handle the DNS failover.

Abstract:  ServerOrigin currently maintains critical infrastructure in two suites at 600 W. 7th St, Los Angeles, CA: Suite 420 (Peer1) and Suite 130 (TelX). Over the course of one week critical infrastructure and some equipment leased to customers will be consolidated into Suite 130. There will be downtime associated with this maintenance; the exact amount will vary from customer to customer with a total impact of 5 – 30 minutes per customer for the entire week.

This is a preliminary notice and there may be changes to the following timeline as we move closer to the maintenance window:

Chronology:

05 DEC 2010, 3PM – 10PM, U.S. Pacific: Many cloud hosted services such as DDoS protection proxies and all Cloud VM instances will be offline during a physical equipment move within this window not expected to exceed 30 minutes. Any such instances should not experience further substantial issues outside of this window.

06 DEC 2010, 12PM – 4PM, U.S. Pacific: There will be edge router maintenance during this period not expected to result in service impact.

07 DEC 2010, 8AM – 10PM U.S. Pacific: There will be a substantial amount of routing changes and physical relocation of edge routers and DDoS mitigation appliances. Service impact is not expected, however, this change is considered high risk and unexpected outages could occur. Our Telia transit will be offline during currently undetermined periods due to local loop maintenance.

08 DEC 2010, 7AM – 5PM: Many customer leased servers will be relocated, resulting in a service impact not expected to exceed 30 minutes per server.

09 DEC 2010, 7AM – 5PM: Many customer leased servers will be relocated, resulting in a service impact not expected to exceed 30 minutes per server.

10 DEC 2010, 8AM – 10PM U.S. Pacific: This window is reserved for final routing changes, network testing, and quality assurance. Service impact is not expected at this time.

11 DEC 2010, 8AM – 10PM U.S. Pacific: This window is reserved for quality assurance and last minute changes. Service impact is not expected at this time.