ItemBank Performance Partial Outage
Incident Report for Illuminate Education
Postmortem

Reported Symptoms: Starting at approx 5am PT, users reported an inability to create new Item Bank test rosters. Additional reports came in that scores from assessments taken were not visible in DnA and that some reports were missing data or failed to load. During this time, any assessment created or scheduled before approx 11pm PT worked normally and students were able to take assessments during this window.

Cause: An Illuminate developer made a change to a message relay component at approximately 11pm on Wednesday April 27th which caused it to become unresponsive to new messages for a period of approximately 3.5 hours on Thursday, April 28th . The delayed messages paused the movement of test scores from the Online Testing app to DnA, prevented the creation of new assessment rosters, and caused some reports to either show missing data or fail to run. The Illuminate engineering team identified the root cause and restored normal functioning of the message subsystem which then allowed all queued messages to move including scores for any assessments taken during the 3 hour window.

Remediation: We have increased our controls around the process for changes to the production messaging subsystem as well as increased monitoring to prevent future inadvertent changes such as this.

Data Loss: No.

We sincerely apologize to any customers who were impacted by this incident. We are committed to continuous improvement of our Online Testing experiences.

Posted 6 months ago. Mar 29, 2019 - 11:24 PDT

Resolved
This incident has been resolved.
Posted 6 months ago. Mar 28, 2019 - 09:17 PDT
Monitoring
[IMPORTANT UPDATE]
Both Flex and Itembank Assessments can continue to be administered via Online Testing.
Rosters can once again be created.
There is no indication any student data was lost during this interruption in Itembank service.
Posted 6 months ago. Mar 28, 2019 - 08:46 PDT
Identified
The underlying issue affecting Itembank performance has been identified and a fix is being implemented.
Posted 6 months ago. Mar 28, 2019 - 08:01 PDT
Update
The team is continuing to investigate the issue.

Flex Assessments can be administered via Online Testing but data cannot be viewed in DNA.

For Itembank Assessments, no new rosters can be created; however, if the roster was already created, then the assessment can be administered. No data can be viewed from Itembank Assessments in DNA.
Posted 6 months ago. Mar 28, 2019 - 07:53 PDT
Update
The team is continuing to investigate the issue.

Flex Assessments can be administered via Online Testing but data cannot be viewed in DNA.

For Itembank Assessments, no new rosters can be created; however, if the roster was already created, then the assessment can be administered. No data can be viewed from Itembank Assessments in DNA.
Posted 6 months ago. Mar 28, 2019 - 07:23 PDT
Update
The engineering team is continuing to investigate.

At the time of this notice:
Flex Assessments can be administered via Online Testing but data cannot be viewed in DNA.

For Itembank Assessments, no new rosters can be created; however, if the roster was already created, then the assessment can be administered. No data can be viewed from Itembank Assessments in DNA.
Posted 6 months ago. Mar 28, 2019 - 06:51 PDT
Update
We are continuing to investigate this issue. Assessments with Itembank items are impacted by this performance issue.
Posted 6 months ago. Mar 28, 2019 - 06:17 PDT
Investigating
The engineering team is currently investigating reports of degraded performance with Itembank.
Posted 6 months ago. Mar 28, 2019 - 05:49 PDT
This incident affected: Illuminate Application (Itembank).