Summary
Earlier today we experienced a service disruption after deploying a database migration related to email handling. The migration triggered a failure in a background subprocessor responsible for synchronizing database views used by the Productlane client. This caused a schema mismatch between the client and server and resulted in errors in the Productlane interface. No data was lost during the incident. All incoming events and messages were safely queued and processed once the system recovered. The issue was fully resolved at 9:12 PM.

We are terribly sorry for the experience this caused.

Timeline

1:10 PM
We deployed a database migration related to emails.

1:14 PM
The migration attempted to create a unique index that included a nullable field. The database rejected this because unique indexes cannot be created on nullable columns in this configuration.

1:18 PM
We removed the index and redeployed the migration.

1:27 PM
Shortly after the migration, one of our background subprocessors responsible for syncing database views began failing. This component runs as part of our infrastructure built on Zero and keeps client-side views in sync with the database.

2:05 PM
Because the view synchronization process failed, the system entered an inconsistent state where the server schema had changed but the client schema had not updated accordingly. This resulted in frontend errors due to the schema mismatch.

3:12 PM
We investigated the failure and identified that the issue originated inside the view synchronization layer. The underlying problem was difficult to debug because the failure occurred inside a core infrastructure component and was not directly visible from the migration itself.

4:36 PM
We prepared a mitigation by restoring compatibility between the client and server schema.

6:18 PM
We deployed the fix which downgraded the client schema to match the server state again, restoring compatibility and resolving the frontend errors. The application was back up, but was not able to write data, still due to the schema mismatch.

6:48 PM
We decided to wipe and rebuild the entire Replica database, which we wanted to avoid, as it can take hours to rebuild.

9:12 PM
All services are restored.

Root cause

The incident was caused by a combination of a database migration that introduced an invalid index configuration and a bug in the view synchronization layer running on top of Zero. While the migration itself was routine, it triggered a failure in this subsystem that caused the view sync workers to stop processing updates. Because the client views rely on this system to stay in sync with the database, the result was a schema mismatch between client and server.

What we will improve

We are taking several steps to reduce the likelihood and impact of similar incidents in the future. We will add more redundancy to our infrastructure, ensuring that failures in individual subprocessors or sync workers cannot interrupt the system.

In addition, we are speeding up our deployment pipeline and introducing instant rollback capabilities so that problematic deployments can be reverted immediately.

We again sincerely apologize for the disruption and appreciate your patience.

After several attempts, we have identified the root cause of the issue and are deploying a fix

We’re currently experiencing a temporary service disruption that may affect parts of Productlane. Our team is investigating the issue and working to restore full service as quickly as possible.

We’ll share another update as soon as we have more information. You can also follow the current status at our status page if needed.

Thanks for your patience while we resolve this.