Outage of German Services

Now that our services are back online, some vulnerabilities in our infrastructure have become much clearer. The sudden power outage and the subsequent power surge upon restoration caused voltage spikes in older server power supplies, which the affected UPS systems were no longer able to fully absorb. This resulted in hardware failures on a large number of servers, particularly when the hard drives restarted simultaneously after power was restored. The issue affects a large portion of the older host systems, the following are affected: col1-compute-amd-1 through col1-compute-amd-1003: These older models had power supplies and controllers that were more vulnerable due to their age and, as a result, suffered massive hard drive failures. col1-compute-amd-1004 through col1-compute-amd-2005: These more modern servers feature improved protection circuits and survived the incident unscathed. col1-compute-intel-1 through col1-compute-intel-25041: The Intel systems are equipped with a different power supply and power distribution architecture that was immune to these specific inrush current spikes. We sincerely apologize for the inconvenience this has caused. Customers with an active backup option are not affected by this data loss, as our backups are stored on external systems and remain intact. According to our Terms and Conditions, backups for hosting-only systems are not included by default. Since we manage a total of more than 5 PB of storage, this service is cost-intensive and not part of the standard package. Corresponding backup options can be added to the server at any time for an additional charge of 20%. Due to the issue, some servers were deleted. Any server that does not reappear by tomorrow at 22:00 (German Time) was unfortunately affected by this data loss, and all affected cloud servers wont be shown in the dashboard anymore. To keep services running, all customers will be moved to temporary servers, which will then be replaced tomorrow. As a result, there will be a brief downtime of approximately 2–4 hours. We will announce the exact timing tomorrow at least 4 hours before the migration begins. You can order new servers tomorrow and restore your data from your own backups. Your previous IP addresses will remain reserved in your account for 48 hours and can be reassigned to your new servers via Support. Naturally, you will not be charged for any downtime, and we are waiving all server runtime costs for July 2026 for affected customers (reflected on your August 1, 2026 invoice) Note: New servers will most likely be restored on our intel servers, since they are not effected by the outage.

Most of the servers are now fully accessible again. Some servers went offline due to simultaneous creation, and a new IP address has been assigned. Please contact our support team if you have been assigned a new IP address, all old addresses remain linked to your account for 48 hours.

All internal services are now fully available again. We are currently working on bringing the cloud’s host systems back online, with SLA customers being our priority. This process will take some time, as we are thoroughly checking each server’s hardware, a sudden power outage can lead to serious errors.

We just received the information that electricity is going to be recovered now, we are monitoring everything to turn on everything asap

Following a minor legal dispute with the city, we were assured that we would be able to re-enter the building this week and that power would be restored.

Unfortunately, we still haven't received any official information from the authorities regarding the outage, so our hands are still tied. In collaboration with our ISP, OVH, we've now been able to get our website back online, but images aren't loading, and unfortunately, we're also unable to manage our VPS due to the outage.