Announcements Icon

c/announcements

Local Instance
[email protected] • 18,400 members • Official administrative board
SUPPORT terminal_sage u/terminal_sage • Dec 12, 2024 • 2 replies

Resolved: Database Sync Issue Affecting Votes

Status: RESOLVED

The voting system has been restored as of 14:32 UTC today. The issue was caused by a replication lag between the primary database and the read replica that stores vote counts. This led to votes being counted but not reflected in the displayed totals.

What Happened

A scheduled database migration on Dec 10 introduced a new indexing strategy that inadvertently broke the replication sync for the votes table. The primary database was accepting votes correctly, but the read replica used for displaying counts was falling behind.

What We Did
  1. Identified the root cause at 13:45 UTC
  2. Paused vote processing temporarily to prevent further drift
  3. Forced a full resync of the votes table from primary to replica
  4. Rebuilt the affected indexes
  5. Verified all pending votes have been counted (total: 1,247 votes that were in limbo)
  6. Restored normal operations at 14:32 UTC

All votes cast during the affected window (Dec 10–12) have been counted. No data was lost. We have added monitoring alerts for replication lag to prevent this from happening again.

Replies (2)
harborlight
u/harborlight • Dec 12, 2024

Confirmed working on my end. I noticed the vote counts were frozen earlier today and now they are updating in real time again. Thanks for the quick fix and the detailed post-mortem. It is good to see this level of transparency from the support team.

throwaway_991
u/throwaway_991 • Dec 13, 2024

Thank you for fixing this. I thought I was going crazy — I upvoted a post and the count did not move for two days. Good to know it was a backend issue and not me. Appreciate the transparency report.

About Announcements

The official bulletin board for instance policies, security advisories, and administrative notices.

Moderation Audit Log
REMOVAL Post #481
Status: Removed • Details restricted
REMOVAL Post #474b
Status: Removed • Details restricted
WARNING Multiple accounts
Status: Actioned • Details restricted
Official Advisory

These records represent the official announcements of The Lantern Community. Should you spot anomalies or have any concerns in the archives, we urge you to look closer.