As some of our customers may have been aware, there was an outage this morning with our website between 5:09am – 9:16am. During this time our website, and subsequently NodePanel, suffered outages. Customer servers continued to operate, but control of servers via NodePanel was mostly inaccessible.
Outage Details
We are currently in the process of switching our automation from the current existing platform to the new platform we’re building NodePanel V2 from. The network backbone and API server were both not yet completely migrated to the new platform, and subsequently did not automatically restart with a system crash as we had anticipated. Simply put, we didn’t restart both servers correctly, leaving our network backbone offline.
Future Prevention Mechanisms
We’ve pushed several changes to the production servers this morning to fully implement them with our new platform, which will catch crashes and automatically restart the server. We’ve also implemented a fix to our web server so that API calls which timeout or error are now displayed to the user rather than ejecting them from a session. With these two patch fixes in place we should be able to better report outages directly to customers and have a much faster time to restart should there be any issues in the future.
Our new release of NodePanel V2 will have a much improved policy for how failures are handled with a fully redundant network backbone. Our API servers will be automatically re-deployed in the instance of any crashes, which ultimately reduces the potential of outages such as this one from lasting any longer than a couple minutes.
In Closing
We feel it’s important for us to be open and transparent regarding these types of issues. As we’ve mentioned before in the past: we want our customers to celebrate the victory we share when things go well, but we also feel that we owe it to them to share our failures as well, as without this, there is no trust in that customer-provider relationship. We hope to keep these type of posts to a minimum as we have over the last few years.
We appreciate our customers patience and understanding in these matters. We expect to keep these types of growing pains to a minimum as we continue to deploy our new platform over the next few months.