An expired certificate and outage led to an undercounting of COVID-19 cases reported in California after 250,00-300,000 lab results were prevented from being uploaded to California's CalREDIE reporting system.
CalREDIE is a data system created by California to report and monitor cases of infectious disease. Using this system, California can more easily spot outbreaks and incidents of community spread as it moves forward with the plans to open schools.
On July 25th, a data outage prevented the CalREDIE system from accepting lab results from external partners. A temporary fix was put into place but was not removed properly, which led to further problems.
In addition to the outage, a certificate expired that prevented lab partners like Quest from uploading California lab results to the data system.
These problems ultimately led to a backlog of 250,000-300,000 results and an underreporting of COVID-19 cases.
As certificates are installed by server administrators once every two years and do not require any upkeep, it could lead to certificates being forgotten about and expiring.
This lack of attention can lead to outages, such as what we saw with Facebook's Tor server, Microsoft Teams, and IoT devices.
“SSL/TLS connections are a client/server protocol and can have two types of certificates; all of them have a server-side certificate that secures the connection and gives browsers some assurance that they’re talking to the right website. But these connections can also have client-side certificates that are used to mutually authenticate the client that initiated the connection. These ‘client’ certificates are becoming more and more prevalent in IT environments with the explosion of DevOps, microservices, cloud architectures and IoT. They often outnumber their traditional server-side counterparts by a factor of 1,000 or more but are often a ‘blind spot’ in an organization, as most traditional cert management tools focus almost exclusively on server-side certs.” said Ted Shorter, CTO of secure digital identity management firm Keyfactor.
While the expired certificate played a part in CalREDIE's problems, California has stated that the system is not designed for the volume of data it is now handling.
In a news conference last week, CHHS Secretary Dr. Mark Ghaly states that California is accelerating the creation of a new reporting system that can handle the increased loads.