Issues accessing Illuminate
Incident Report for Illuminate Education

Yesterday, at approximately 3:10 a.m. Pacific Daylight Time (PDT), Illuminate Education's internal automated monitoring systems alerted us to external connectivity issues between our primary hosting operations facility and Internet. Illuminate Education systems remained up and operational, however, customers reported issues accessing Illuminate Education from the internet.

At Illuminate Education, maintaining system availability is a top priority. When we were alerted of the issue, we immediately deployed a team of engineers to resolve the issue, which was determined that a hardware failure had occurred. The primary interface between our internet service provider (ISP) and the Illuminate Education environment had failed. This connection has redundancy, however the failover did not consistently route all traffic around the failed component, resulting in the inconsistent access to Illuminate Education.

Our engineers resolved the issue at approximately 2:29 p.m. PDT. At that time, full access to all system services was restored. The engineers verified that all Illuminate core services were available and operating normally. At no time was customer data lost or at risk.

As we maintain replacement hardware on site, our team was able to expeditiously replace the failed component. Our automated failover system was also corrected, which will prevent such an issue from impacting customers in the future.

Illuminate Education takes this issue very seriously and we want to personally apologize for any inconvenience that may have been caused by this issue. We are conducting an internal review of our hosting operations to ensure we are doing everything possible to maintain the high levels of system uptime you expect from us.

If you have any questions or concerns, please do not hesitate to contact us at (949) 242-0343.

Posted about 1 month ago. Apr 14, 2017 - 14:28 PDT

Resolved
Our engineers have identified the core issue and has resolved it. All users are once again able to access all areas of Illuminate. A full write-up on this incident will be posted shortly.
Posted about 1 month ago. Apr 13, 2017 - 17:09 PDT
Update
Our engineers are continuing to work to isolate and resolve the underlying issue. Resolving this issue is our top priority.
Posted about 1 month ago. Apr 13, 2017 - 14:22 PDT
Update
We are continuing to investigate the network connectivity issues and are continuing to conduct diagnostics with our hosting operations provider. We are working to resolve this issue as soon as possible.
Posted about 1 month ago. Apr 13, 2017 - 12:03 PDT
Update
We are continuing to investigate connectivity issues with our hosting operations provider. As part of the troubleshooting process, we are utilizing diagnostics which could briefly interrupt system access. We will do everything possible to minimize the impacts of these diagnostics. Resolving this issue is our top priority.
Posted about 1 month ago. Apr 13, 2017 - 10:37 PDT
Update
After further investigation, it appears that the issues our users are experiencing could be related to issues with our 3rd party networking and ISP partners. We are currently engaging their support teams in order to determine the root cause of the outage.

Until this outage is resolved, users will continue to experience sporadic connectivity issues with Illuminate and Illuminate HC. Due to the nature of the underlying issue, users may also experience issues with modules and functionality.
Posted about 1 month ago. Apr 13, 2017 - 08:57 PDT
Update
We apologize for the ongoing outage. Many users are still reporting sporadic issues with logging in Illuminate and Illuminate Home Connection.
Posted about 1 month ago. Apr 13, 2017 - 06:18 PDT
Identified
Some districts are reporting intermittent issues when trying to access Illuminate. We have confirmed that an issue exists and our engineers are working towards a solution.
Posted about 2 months ago. Apr 13, 2017 - 05:02 PDT