Loading...

Category Archives for System Notices

Update on System Outage, 4/29/14

Late last night Voter Gravity suffered a system-wide outage due to a database failure. Our engineering team responded to the event immediately and worked with the Amazon Web Services team throughout the night to restore the database. Early this morning, full functionality of the system was restored. We are still examining the database, but we have no reason to believe there has been any loss of user data. We will continue to investigate to be 100% certain.

The outage resulted in users being unable to access the system from approximately 9:30 pm ET 4/29 through 5:30 am ET 4/30. We sincerely apologize for this issue and the impact it had on our users. As political operatives ourselves, we know how critical it is that you have all systems go 24/7 on a campaign. That’s why we’ve already started to implement a mitigation process to ensure this outage never happens again.

Here are the specific steps we are taking immediately:

1) In addition to the monitoring and alert tools we have in place, we are setting up additional alerts to catch incidents even earlier.

2) We are re-evaluating our database redundancy and availability strategy to be better prepared to switch quickly in case there is ever a future failure, taking the potential impact from hours to seconds.

3) We have been in the process of making several major database changes to further optimize our system. We will accelerate our work on these projects as they will not only make our core infrastructure more robust, but improve the quality of the user experience as well.

4) We are adding additional user notifications in the portal and via status updates. We will have more news on these alerts and notifications in the near future.

Again, we deeply apologize to any of our users affected by this incident. If you have any questions or notice any issues with your account, please don’t hesitate to contact our Support Team.