Update on December 9 Outage

Following yesterday's outage, I wanted to take a few minutes to give you a bit more background about what happened, and what our team is doing to address the situation. Before diving in to the details, I want to confirm for you that:

  • Your data is (and always was) secure. The outage was not related to a security breach, and no data was lost during the process of bringing our servers back online.
  • I know outages are frustrating and disruptive. We strive to maximize the uptime of our systems with robust testing, monitoring and reporting; however,  despite those safeguards, we experienced downtime this week.

That being said, here are the details about this week's events.

Why did this happen?

Like many other companies, we rely on Amazon to provide some elements of our technical infrastructure. Their tools are powerful and are key to letting us scale our systems to meet your needs.

On Tuesday, Amazon experienced an outage that caused us to lose connectivity to our primary database cluster (this is where all of your data is stored). This loss of connectivity broke the database cluster's ability to index and search for Tags, which many of our customers use to organize their data in Method.

While the tag data itself was untouched, the ability to search these tags was corrupted. After our systems came back online from this initial outage, the database cluster attempted to repair the tag indexes. The corruption caused the number of processes attempting to repair the indexes to increase, ultimately getting to a point where the database cluster was overwhelmed and was not able to accept any new connections. This is what led to yesterday's outage.

How did you fix the problem?

We were able to bring the databases back online by disabling the database process that indexes and makes tags searchable. This allows you access back into Method, while we work behind the scenes to fix the broken processes and reindex the tags. Part of the complication yesterday was that this functionality was relying on an obscure 3rd-party feature that is not well documented. As our team was troubleshooting this issue, they were literally writing a manual for it on the fly as they tested various fixes. 

Where do we go from here?

Our team is working non-stop to restore tag functionality. As of right now, tags created before December 8th are searchable and function as expected. Any tags created on or after that date are not searchable yet. We will be testing and deploying fixes with the goal of restoring full functionality prior to the start of the workday on Monday.

In parallel, we are also investigating alternative solutions for the indexing of tags so that we can avoid encountering this specific issue again in the future.

While the overall issues we have faced were initially triggered by one of our upstream providers, we know that you rely on Method to be available and reliable for when you and your team want to use it. Our commitment to providing maximum uptime has not changed - and we'll continue to strive to keep you informed and up-to-date during outages via our status page.

If you have any questions or concerns, please open a conversation with our team via live chat or by sending an email to support@method.me. Thank you for choosing Method - we truly value you and the trust you place in our team and platform to help you grow your business. 

Sincerely,

Paul Jackson
CEO & Founder