Update: Monday 21st April 2008 – 10:48BST
Clearly it all went badly wrong. The “worst case scenario” was exceeded by 50%.
We will be writing to customers later today with a full explanation.
Why is this happening?
Some customers have asked why this is happening. Isn’t Level3 is supposed to be a resilient facility?
The reason is well known in the Data Center industry. There is a massive shortage of data center capacity in the UK (in particular London).
The problem is not space, but power. High density servers like blades combined with massive growth in demand for data center services has meant that large data centers are being forced to hastily add power capacity.
This is exactly what is happening with Level3’s Braham Street facility (the one in question).
This data center is less than a decade old yet its power systems cannot keep up with demand. To stay in business and accommodate growing power and cooling demands, Level3 and other facility providers are having to add capacity.
The good news is that this is the sting for those of us using Level3’s facility. Just about every facility is being forced to do this, leading to high profile power related downtimes for the likes of PayPal , MySpace, Craigslist and others.
Original Notification
What is happening?
We have received notification from our Upstream and Data Centre provider (Level3) that they will be carrying out “Service Affecting Maintenance” on the 19th of April 2008.
They are adding additional UPS capacity to their data centre as part of a 5 yearly maintenance and upgrade plan.
As a consequence of this upgrade, all power at the Level3 facility will be switched off.
What does it mean?
It means that for several hours on the 19th April 2008 there will be no power to any racks in the Level3 facility. All equipment hosted at Level3 will be down. The outage will last approximately 6 hours.
During the outage website, servers (including DataGate DNS servers) and mail will all be unavailable. Mail sent to you will NOT be bounced. All messages will spool in a queue until the services are restored.
We are extremely sorry about all this.
DataGate Engineers will carry out controlled power downs of all DataGate and managed customer equipment to ensure that there is no damage due to sudden loss of power.
What can you do?
User intervention for the Shared Web Hosting, Exchange Hosting and Managed Dedicated Servers customers is not required.
DataGate Engineers will gracefully power down equipment and make sure that data is safe. When power is restored we will revive all services and check that all systems are fully operational.
Customers who have Dedicated Servers but without a managed services agreement (Monthly Administration package) are required to:
- Power down their server(s) gracefully before 07:15AM on 19th of April
- Make sure any start up scripts and/or programs are in place and set to automatically activate on the server after the servers are powered on again.
These measures are for your own benefit as sudden power cuts can be extremely damaging to servers.
What will happen next?
We will be sending you an update in the next few days in case that we have any changes of exact times and estimates of our power-down schedule.
If you have any questions at all about this downtime schedule, we have activated a special support hotline number and e-mail address::
+44(0)208 099 3504
noc-support AT datagate.net.uk.
DataGate managers will be available from 08:30AM GMT to 15:30PM GMT Mon-Fri to answer any queries and help you with any problems related to this outage.
Finally, please accept our sincerest apologies for the trouble this will cause you. If there is any way we can help mitigate problems arising from this please let us know at the address or number above.
Posted by datagate 