No incidents reported this month.
No incidents reported this month.
- InvestigatingNovember 20, 2021 at 2:21 PMInvestigatingNovember 20, 2021 at 2:21 PM
We are aware that an error internally has led to incorrect suspension emails being sent out. We are currently working on resolving this as soon as possible.
We deeply apologise for any inconvenience this may have caused.
- IdentifiedNovember 20, 2021 at 4:40 PMIdentifiedNovember 20, 2021 at 4:40 PM
It seems like some services were suspended nonetheless. We've identified the issue and are applying a fix. We're currently working on getting services back up and running that are suspended.
- IdentifiedNovember 20, 2021 at 6:41 PMIdentifiedNovember 20, 2021 at 6:41 PM
We're all hands on deck to resolve the issues as quickly as possible. Some of the internal systems caused some unexpected suspensions. The good news is that we have backups so we're working on restoring data as quickly as possible.
- IdentifiedNovember 20, 2021 at 8:26 PMIdentifiedNovember 20, 2021 at 8:26 PM
The backup restore on all services has started. Due to the amount of services that need to be restored. This process might take a while.
We will notify about status updated here.
- ResolvedNovember 20, 2021 at 10:16 PMResolvedNovember 20, 2021 at 10:16 PM
We've restored all backups and things should be working as expected. Obviously, due to the scale of this operation, we will be monitoring services for the next few days. If you experience any issues feel free to contact us via ticket and we will assist you as quickly as possible.
A post mortem will be sent out within the next few days along with compensation for the affected customers.
Thank you for your understanding. We will do everything within our power to prevent issues like these from happening in the future.
- ResolvedNovember 24, 2021 at 10:04 PMResolvedNovember 24, 2021 at 10:04 PM
As Dashflo continues to grow, we put great value in being transparent within our community. When negative situations arise that could possibly affect our customers and their services, we want to be as clear as possible about the situation and our reaction.
Last weekend, such an incident took place and below is a summary of our actions.
This incident affected the following services:
- Web Hosting
- Minecraft Hosting
- Discord Bot Hosting
Starting at 3:21 PM CET on November 20 2021, some customers of web hosting, discord bot hosting and game hosting services encountered the suspension of their services due to an internal error in our systems.
At 3:25 PM CET, we became aware of the internal error that led to services being suspended. At this point, we also came to the conclusion that a very limited number of services were terminated.
Issue & Fix
At 3:47 PM CET we identified the root cause of the issue and were able to prevent any further unwanted actions from taking place. Alongside that, we were actively working on unsuspending affected services. This work consisted of creating automated processes that would kick off the full restoration of services.
At 9:26 PM CET, the restore on all services was put in motion but needed time taking into consideration the number of services we are dealing with.
At 11:16 PM CET we had brought all our services to roughly 99% functionality. From this point, the situation was 100% under control and being continuously monitored.
If your services were affected by this outage, please reach out to us by opening a ticket to our billing department. We will apply a compensation of 14 days to each service that was affected by this outage.
If there are any issues, please feel free to contact us.
What went well
Monitoring alerted us quickly Rapid response from our engineering team We were focused and continued to update our customers and status page.
What didn’t go so well (lesson learned)
It took us far too long to identify what customers were affected. The restore from backups was too slow for certain services, such as web hosting.
The incident itself was caused by an unwanted side-effect of our power maintenance in our Frankfurt datacenter that occurred earlier that day. Within minutes of the incident, our entire team was working on finding the cause. While we do have procedures prepared for many scenarios, this was a truly unique one which is why it took us some time to start service restoration.
During the next few weeks, we will be putting in a considerable amount of work to optimize our recovery and restoration procedures. While we were still fairly well prepared, an incident like this allows us to learn and improve on what we already have.
Ajdin L - Director
- PlannedNovember 21, 2021 at 4:00 AMPlannedNovember 21, 2021 at 4:00 AM
The node game-001.dfw will undergo a reboot to perform a brief maintenance. Your service will experience a short interrtuption. We apologize for the inconvenience, and appreciate your cooperation.
- CompletedNovember 21, 2021 at 4:04 AMCompletedNovember 21, 2021 at 4:04 AM
The maintenance has been completed successfully. Thanks for bearing with us.
- InvestigatingNovember 27, 2021 at 1:03 PMInvestigatingNovember 27, 2021 at 1:03 PM
We are currently investigating connectivity issues in our Frankfurt datacenter. We will update when we have more information
- ResolvedNovember 27, 2021 at 1:19 PMResolvedNovember 27, 2021 at 1:19 PM
The issue was with our upstream provider. We will continue to monitor the network.
- PlannedNovember 28, 2021 at 5:30 PMPlannedNovember 28, 2021 at 5:30 PM
We will be performing maintenance to our US infrastructure. Due to this, US game nodes will experience momentary unavailability lasting a few minutes.
- In progressNovember 28, 2021 at 5:30 PMIn progressNovember 28, 2021 at 5:30 PM
Maintenance is now in progress.
- CompletedNovember 28, 2021 at 5:38 PMCompletedNovember 28, 2021 at 5:38 PM
Maintenance has been completed successfully.