Skip to main content

One post tagged with "data recovery"

View All Tags

Database Migration Incident Summary

· One min read
Mike Thrift
Mike Thrift
Marketing Manager

Incident summary​

On 2021-08-03 2:35pm PST, one of our engineers made a bad database migration that caused discrepancies between the indexed data and the source of truth in the database. It impacted 39 users, and we backfilled the data and resolved the issue at 4:46pm PST.

Impact​

2021-08-03-incident-2021-08-03

Those impacted 39 users may lose data added between 2021-08-03 2:35pm PST and 4:46pm PST. We backfilled the data but cannot guarantee 100% recovery.

Root cause​

The root cause is our new database migration to reorganize the file structure and prepare for the dropbox integration. Unfortunately, we underestimated the number of users visiting this service during the deployment.

Lessons learned​

Next time in similar situations, we will

  1. Be more cautious about the database migration. Be aware that there are data insertions during the migration.
  2. Set the site to maintenance mode when we need to stop all the traffic and racing conditions.

About Beancount.io

Beancount.io is a powerful, open-source double-entry accounting system trusted by finance professionals, developers, and personal finance enthusiasts alike. Designed for clarity, extensibility, and precision, Beancount helps users manage complex financial data with plain-text accounting, robust validation, and seamless integrations. Since its inception, it has empowered users to gain full control over their finances, automate reporting, and maintain transparent, auditable records—making it a favorite among those who value simplicity without compromise.