Reported Symptoms: Starting at approx 5am PT, users reported an inability to create new Item Bank test rosters. Additional reports came in that scores from assessments taken were not visible in DnA and that some reports were missing data or failed to load. During this time, any assessment created or scheduled before approx 11pm PT worked normally and students were able to take assessments during this window.

Cause: An Illuminate developer made a change to a message relay component at approximately 11pm on Wednesday April 27th which caused it to become unresponsive to new messages for a period of approximately 3.5 hours on Thursday, April 28th . The delayed messages paused the movement of test scores from the Online Testing app to DnA, prevented the creation of new assessment rosters, and caused some reports to either show missing data or fail to run. The Illuminate engineering team identified the root cause and restored normal functioning of the message subsystem which then allowed all queued messages to move including scores for any assessments taken during the 3 hour window.

Remediation: We have increased our controls around the process for changes to the production messaging subsystem as well as increased monitoring to prevent future inadvertent changes such as this.

Data Loss: No.

We sincerely apologize to any customers who were impacted by this incident. We are committed to continuous improvement of our Online Testing experiences.