Users were unable to login to the Illuminate application for approximately one hour on April 22 due to a change in the way tumblr.com handles API requests. At approximately noon on April 22, tumblr.com stopped processing requests to http addresses. As a result of this change, a component of the main login page could not be created and thus generated a critical error within the application. The critical error prevented the login page from being rendered. Users who were logged into the system prior to the change in behavior from tumblr.com were not impacted.
The engineering team deployed a hotfix to submit proper requests to tumblr.com. Once the hotfix was deployed, the application cache (on the server side, not in the user’s browser) was cleared. The total duration of the outage varied between 45 minutes to one hour depending on the cache clearing sequence employed by the Hosting Operations team.
As a result of this incident, the engineering team is taking steps to ensure our requests to external dependencies, such as tumblr.com, are no longer capable of blocking functionality of the application if or when they fail.