All systems operational

Incident history

Nov 2021 to Jan 2022

Jan 2022

No incidents reported this month.

Dec 2021

No incidents reported this month.

Nov 2021

Accidental service suspensionsResolved after about 8 hours
  • Investigating
    November 20, 2021 at 2:21 PM

    We are aware that an error internally has led to incorrect suspension emails being sent out. We are currently working on resolving this as soon as possible.

    We deeply apologise for any inconvenience this may have caused.

  • Identified
    November 20, 2021 at 4:40 PM

    It seems like some services were suspended nonetheless. We've identified the issue and are applying a fix. We're currently working on getting services back up and running that are suspended.

  • Identified
    November 20, 2021 at 6:41 PM

    We're all hands on deck to resolve the issues as quickly as possible. Some of the internal systems caused some unexpected suspensions. The good news is that we have backups so we're working on restoring data as quickly as possible.

  • Identified
    November 20, 2021 at 8:26 PM

    The backup restore on all services has started. Due to the amount of services that need to be restored. This process might take a while.

    We will notify about status updated here.

  • Resolved
    November 20, 2021 at 10:16 PM

    We've restored all backups and things should be working as expected. Obviously, due to the scale of this operation, we will be monitoring services for the next few days. If you experience any issues feel free to contact us via ticket and we will assist you as quickly as possible.

    A post mortem will be sent out within the next few days along with compensation for the affected customers.

    Thank you for your understanding. We will do everything within our power to prevent issues like these from happening in the future.

  • Resolved
    November 24, 2021 at 10:04 PM

    As Dashflo continues to grow, we put great value in being transparent within our community. When negative situations arise that could possibly affect our customers and their services, we want to be as clear as possible about the situation and our reaction.

    Last weekend, such an incident took place and below is a summary of our actions.

    This incident affected the following services:

    • Web Hosting
    • Minecraft Hosting
    • Discord Bot Hosting

    Incident

    Starting at 3:21 PM CET on November 20 2021, some customers of web hosting, discord bot hosting and game hosting services encountered the suspension of their services due to an internal error in our systems.

    Leadup

    At 3:25 PM CET, we became aware of the internal error that led to services being suspended. At this point, we also came to the conclusion that a very limited number of services were terminated.

    Issue & Fix

    At 3:47 PM CET we identified the root cause of the issue and were able to prevent any further unwanted actions from taking place. Alongside that, we were actively working on unsuspending affected services. This work consisted of creating automated processes that would kick off the full restoration of services.

    Recovery

    At 9:26 PM CET, the restore on all services was put in motion but needed time taking into consideration the number of services we are dealing with.

    At 11:16 PM CET we had brought all our services to roughly 99% functionality. From this point, the situation was 100% under control and being continuously monitored.

    Compensation

    If your services were affected by this outage, please reach out to us by opening a ticket to our billing department. We will apply a compensation of 14 days to each service that was affected by this outage.

    Support

    If there are any issues, please feel free to contact us.

    What went well

    Monitoring alerted us quickly Rapid response from our engineering team We were focused and continued to update our customers and status page.

    What didn’t go so well (lesson learned)

    It took us far too long to identify what customers were affected. The restore from backups was too slow for certain services, such as web hosting.

    Root cause

    The incident itself was caused by an unwanted side-effect of our power maintenance in our Frankfurt datacenter that occurred earlier that day. Within minutes of the incident, our entire team was working on finding the cause. While we do have procedures prepared for many scenarios, this was a truly unique one which is why it took us some time to start service restoration.

    During the next few weeks, we will be putting in a considerable amount of work to optimize our recovery and restoration procedures. While we were still fairly well prepared, an incident like this allows us to learn and improve on what we already have.

    Thank-you
    Ajdin L - Director

Node reboot [game-001.dfw]
  • Planned
    November 21, 2021 at 4:00 AM

    The node game-001.dfw will undergo a reboot to perform a brief maintenance. Your service will experience a short interrtuption. We apologize for the inconvenience, and appreciate your cooperation.

  • Completed
    November 21, 2021 at 4:04 AM

    The maintenance has been completed successfully. Thanks for bearing with us.

Connectivity issues FrankfurtResolved after 17 minutes
  • Investigating
    November 27, 2021 at 1:03 PM

    We are currently investigating connectivity issues in our Frankfurt datacenter. We will update when we have more information

  • Resolved
    November 27, 2021 at 1:19 PM

    The issue was with our upstream provider. We will continue to monitor the network.

US Game Maintenance
  • Planned
    November 28, 2021 at 5:30 PM

    We will be performing maintenance to our US infrastructure. Due to this, US game nodes will experience momentary unavailability lasting a few minutes.

  • In progress
    November 28, 2021 at 5:30 PM

    Maintenance is now in progress.

  • Completed
    November 28, 2021 at 5:38 PM

    Maintenance has been completed successfully.