Monday, 4 Oct a dealer reported that their website was not working when getting to it via a web search. The problem was identified and fixed 90 mins later.

C9 hosted websites with SSL certificates stopped working on some devices, such as iPhones. On these devices the web browser would report that there is a security issue with the website and that the websites SSL certificate had expired.

Root cause: c9 hosted websites use letsencrypt for SSL certificates.  SSL certificates contain a list of certificates in a 'chain of trust'.  The root of letsencrypt certificates expired a couple of days ago and was replaced with a new root.

The problem I believe is that c9's webserver was configured such that it didn't provide the full certificate chain, it only provided the top of the chain.  Most web browsers fill in the gaps using trusted chains they know about. But with transition to a new root, some environments did not have enough information to fill in the gaps and could not validate that the link to the website was secure/trusted.

The fix involved making sure c9 webserver served the full chain of certificates.  The webserver we use made this fiddly to achieve so took a little longer than it ordinarily should of to resolve. One longer term mitigation we will need to reconsider our webserver stack (consider nginx to replace lighttpd).