DWP Hosting is back online

18:50 Update:

We’re pleased to report that all live databases are now back online and available for use. If you notice any issues, please report them immediately to support@pellcomp.co.uk

Thank you for your patience today at this busy time.

16:00 Update:

We are still health-checking data and fixing any issues encountered. Unfortunately, this is a slow process and is running longer than we would have hoped. Please bear with us.

15:00 Update:

Power has now been restored to our equipment, so we’ve temporarily put the Hosting Portal into maintenance mode in order to health check databases before resuming normal service. Barring any major issues, this should be within the hour.

14:00 Update:

We are still waiting for engineers to restore power to our equipment.

13:00 Update:

The Telstra DC suffered a localised fire which has disrupted power to several floors including the floor where our servers are situated. Telstra engineers are on site working on a solution which involves rerouting power from the generators. Pellcomp have initiated our DR procedure as a precaution in case normal service is not resumed today.

Here are the updates from Telstra…

Update 1:
ALERT- LHC (London Hosting Center) team are reporting a fire alarm has been triggered on site at approx. 08:30 UK time, suggestion that it relates to the UPS which supports all of the power on the 3rd floor. We are coordinating with DC support staff to confirm if any of our devices impacted. We are also working with IP and Transmission NOC to assess the impact if any to our customers. Updates to follow.

Update 2:
“Due to a localized fire in the LHC we have lost the Green system. This provides power to the lower half of the building. The fire has tripped the breakers supporting the busbar. Engineers are on site and are working to restore power via generator as we speak. Network room ULHC in 410 has a feed from this system but is unaffected.”

Update 3:
“Level 3 in the building is still down. The network room (Room 410) is on the 4th floor and fed from level 3 but the alternative power supply (blue feed) from the 5th floor is also down, hence the outage. DC Team is continuously working with priority to find out the root cause and fix.”

Update 4:
“As observed some of the IP devices are now resuming and IPNOC are verifying it at this stage.Loss of connectivity is expected until the power fully recovered in ULHC.

Telstra onsite engineers are going through all racks in Network Room 410 to reboot any kit that did not come back automatically.

We will update you as soon as this action is complete”

12:00 Update:

Power is gradually being restored across the facility, so we hope to have contact with our servers soon. It may take some time to health-check all systems and bring databases back online though. Please bear with us – further updates will follow.

11:00 Update:

The cause of the outage is down to a fire at the Telstra London Hosting Centre, which resulted in all power being taken down at the facility. The fire was under control by 9:30, so we’re now just waiting to hear when power can be restored. We’ll update again when we know more.

Original message:

We are currently investigating a major outage affecting users of our DWP Hosting platform. We will provide further information as soon as we know more.

Status updates are always available on our Hosting Status page.

Skills Hosting is unaffected as that runs from a different data centre.