Maintenance in progress

[Resolved] Accidental service suspensions

Started on November 20, 2021 at 2:21 PM. Resolved after about 8 hours

Affected

Services
Game
Services
Discord Bot Hosting
Services
Web Hosting
  • Investigating
    November 20, 2021 at 2:21 PM

    We are aware that an error internally has led to incorrect suspension emails being sent out. We are currently working on resolving this as soon as possible.

    We deeply apologise for any inconvenience this may have caused.

  • Identified
    November 20, 2021 at 4:40 PM

    It seems like some services were suspended nonetheless. We've identified the issue and are applying a fix. We're currently working on getting services back up and running that are suspended.

  • Identified
    November 20, 2021 at 6:41 PM

    We're all hands on deck to resolve the issues as quickly as possible. Some of the internal systems caused some unexpected suspensions. The good news is that we have backups so we're working on restoring data as quickly as possible.

  • Identified
    November 20, 2021 at 8:26 PM

    The backup restore on all services has started. Due to the amount of services that need to be restored. This process might take a while.

    We will notify about status updated here.

  • Resolved
    November 20, 2021 at 10:16 PM

    We've restored all backups and things should be working as expected. Obviously, due to the scale of this operation, we will be monitoring services for the next few days. If you experience any issues feel free to contact us via ticket and we will assist you as quickly as possible.

    A post mortem will be sent out within the next few days along with compensation for the affected customers.

    Thank you for your understanding. We will do everything within our power to prevent issues like these from happening in the future.

  • Resolved
    November 24, 2021 at 10:04 PM

    As Dashflo continues to grow, we put great value in being transparent within our community. When negative situations arise that could possibly affect our customers and their services, we want to be as clear as possible about the situation and our reaction.

    Last weekend, such an incident took place and below is a summary of our actions.

    This incident affected the following services:

    • Web Hosting
    • Minecraft Hosting
    • Discord Bot Hosting

    Incident

    Starting at 3:21 PM CET on November 20 2021, some customers of web hosting, discord bot hosting and game hosting services encountered the suspension of their services due to an internal error in our systems.

    Leadup

    At 3:25 PM CET, we became aware of the internal error that led to services being suspended. At this point, we also came to the conclusion that a very limited number of services were terminated.

    Issue & Fix

    At 3:47 PM CET we identified the root cause of the issue and were able to prevent any further unwanted actions from taking place. Alongside that, we were actively working on unsuspending affected services. This work consisted of creating automated processes that would kick off the full restoration of services.

    Recovery

    At 9:26 PM CET, the restore on all services was put in motion but needed time taking into consideration the number of services we are dealing with.

    At 11:16 PM CET we had brought all our services to roughly 99% functionality. From this point, the situation was 100% under control and being continuously monitored.

    Compensation

    If your services were affected by this outage, please reach out to us by opening a ticket to our billing department. We will apply a compensation of 14 days to each service that was affected by this outage.

    Support

    If there are any issues, please feel free to contact us.

    What went well

    Monitoring alerted us quickly Rapid response from our engineering team We were focused and continued to update our customers and status page.

    What didn’t go so well (lesson learned)

    It took us far too long to identify what customers were affected. The restore from backups was too slow for certain services, such as web hosting.

    Root cause

    The incident itself was caused by an unwanted side-effect of our power maintenance in our Frankfurt datacenter that occurred earlier that day. Within minutes of the incident, our entire team was working on finding the cause. While we do have procedures prepared for many scenarios, this was a truly unique one which is why it took us some time to start service restoration.

    During the next few weeks, we will be putting in a considerable amount of work to optimize our recovery and restoration procedures. While we were still fairly well prepared, an incident like this allows us to learn and improve on what we already have.

    Thank-you
    Ajdin L - Director