On Friday October 11 and Saturday October 12, we had a significant outage that affected Billings Pro subscribers. Two different failures that occurred together made it a miserable experience for our customers. Here is what happened.
The 1st error was a bug in the app. The app itself should continue to work if there is no internet or if there is a failure on our servers. The no internet bit works fine, but when our backend system had a problem, the app misinterpreted that error as an “invalid credentials” message and prompted the user to verify credentials continuously.
The 2nd error was the more serious one. We use PostgreSQL to power the database portion of our backend. PostgresSQL is used by the likes of Skype, Etsy, IMDB and more and is fully capable of scaling for our needs. However, we use it a bit differently (every account has its own set of tables – which is a different technique than most others) and we ran into an esoteric issue as we hit a certain threshold in volume. Diagnosing the issue is what took us the bulk of the time. Once we found the cause, fixing it didn’t take too long. That fix was deployed on Saturday around 11:30am EST. The system has been stable since then.
During the downtime, at no time was your data at risk.
Having fixed the second issue, we are making changes to Billings Pro on Mac in version 1.6.6 and Billings Pro on iOS in version 1.7.3 to address the first issue, so that the interpretation of the error message doesn’t prevent you from using the app.
We sincerely apologize for any inconvenience this event may have caused you. If you have any questions, please email us at info [@] marketcircle.com.