Over reporting of podcast downloads between 13pm and 15pm on March 10
Incident Report for iono.fm
We have developed a script to perform this fix, and this has now been run on our production dataset.

All values for March 10 has been removed and will now start rebuilding without the duplicated values.

We expect all reporting data for yesterday to be available in a few hours.
Posted about 2 years ago. Mar 11, 2016 - 15:27 SAST
At 13pm today a failed upgrade triggered a failure mode that caused duplicate entries in our podcast download stats, leading to inflated download numbers. (Which should be clearly visible on download reports for today) Note this affects only entries capture between 13h and 15h.

Additionally this has led to our reporting lagging behind real-time.

As a resolution to this issue we will clear all data report for today, remove all duplicates and rebuild the report data. While duplicates are easy to detect and this process should not result in any data loss, the consistency of your data is of paramount importance and so we've scheduled this task for tomorrow morning when our development team is fresh.

After clearing today's data, your will notice a gap in your reports, which will get rebuilt from the corrected data over time. We hope to complete this process by close of business tomorrow, after which the data will be rebuilt over the course of a few hours.

We will also update this incident report as we make progress.
Posted about 2 years ago. Mar 10, 2016 - 19:05 SAST