Some users unable to login to Illuminate
Incident Report for Illuminate Education
Postmortem

Users were unable to login to the Illuminate application for approximately one hour on April 22 due to a change in the way tumblr.com handles API requests. At approximately noon on April 22, tumblr.com stopped processing requests to http addresses. As a result of this change, a component of the main login page could not be created and thus generated a critical error within the application. The critical error prevented the login page from being rendered. Users who were logged into the system prior to the change in behavior from tumblr.com were not impacted.

The engineering team deployed a hotfix to submit proper requests to tumblr.com. Once the hotfix was deployed, the application cache (on the server side, not in the user’s browser) was cleared. The total duration of the outage varied between 45 minutes to one hour depending on the cache clearing sequence employed by the Hosting Operations team.

As a result of this incident, the engineering team is taking steps to ensure our requests to external dependencies, such as tumblr.com, are no longer capable of blocking functionality of the application if or when they fail.

Posted Apr 22, 2019 - 17:29 PDT

Resolved
This incident has been resolved. All customers can once again login.
Posted Apr 22, 2019 - 13:21 PDT
Monitoring
The issue has been resolved. Customers are once again able to login to the application. The interruption in service was due to changes made by tumblr.com. Full details will be made available in the postmortem.
Posted Apr 22, 2019 - 13:04 PDT
Identified
The engineering team has identified the root cause preventing some users from logging into Illuminate. A fix is being prepared and will be deployed to production sites in a few moments.
Posted Apr 22, 2019 - 12:40 PDT
Investigating
Engineers are aware of reports from some users unable to login to Illuminate. The team is currently investigating these reports and attempting to identify the root cause.
Posted Apr 22, 2019 - 12:32 PDT
This incident affected: DnA.