I run a breedable pet business in both Second Life and on Enchanted Grid in OpenSim under the name Foxtrot Breedables.
Starting at 6 a.m. this morning, we began getting an influx of tickets.
We found that our OpenSim customers were not affected by the disruption.
The Second Life issue, a security certificate expiration, has not been fixed yet.
OpenSim won this battle of the grids.
What happened with Second Life?
On Thursday, September 30, a 20-year encryption certificate expired in Second Life.
Suddenly all those companies that rely on encrypted communication from the Second Life grid to the World Wide Web came to a halt. What happened? Let’s get into it.
One of the root certificates for the Certificate Authority called “Let’s Encrypt” expired. Let’s Encrypt is a free service that many people use in their everyday lives. It’s considered safe and reliable.
So what happened? It’s the root certificates that Second Life acquired that expired, their time up.
It was a 20-year certificate. That seemed wild to me.
It means that in-world LSL scripts making HTTP calls to websites secured with Let’s Encrypt certificates are failing grid-wide. Second Life is doing the clean-up now, but many businesses like my own were down for many hours on Thursday. As of this writing, it’s still an ongoing issue, and companies are still being impacted.
Foxtrot Breedables operates out of Second Life and the Enchanted Grid. We were one of those impacted businesses today. Enchanted Grid doesn’t use Let’s Encrypt, so their breedable pets were working fine, not an ounce of downtime, a win for OpenSim. Meanwhile, Second Life breeders had no clue what was going on, there was panic and confusion when butterflies made babies that said they were bunnies.
What we’re doing about it
We asked our Second Life breeders to stop all breeding. Our breeders have to do this themselves. It’s not automatic like most breedables.
We checked Enchanted Grid and gave their breeders an all-clear. They were not experiencing the same breaks that the Second Life breeders were. My scripter and I waited for hours until we could hear updates from Second Life. When we did, we grew concerned that this wasn’t going to be an easy fix.
Talking it out with whomever you’re working with, no matter the issues, is always a good thing. So I spoke with my scripter and database admin. I asked if we could change our certificate on our hosting. My scripter suggested it might be a good idea, so we investigated.
I found that our hosting company, DreamHost, offered a Sectigo DV certificate for $15 a year. We figured out that Sectigo was not part of Let’s Encrypt. That’s all it took for me to suggest we change our certificate away from Let’s encrypt, and we had our people up and running in like fifteen min from that decision. Companies like Maitreya closed off their sims due to vendors not being responsive. Vendors like Caspervend and Casperlet were also experiencing issues — which is most likely why the Maitreya, mentioned above, closed their sim access.
We solved the issue in very little time, though it did cost us $15.00 for an entire year of encryption. There is nothing wrong with Let’s Encrypt, but it’s not very reliable for Second Life right now until they get a fix, and if you have customers waiting, this is a good option for you: Change your Certificate away from Let’s Encrypt.
This ongoing Root Certificate issue doesn’t feel like it’ll take hours to fix. It feels like it will take more like days. We know the Linden team is working their butts off to find a solution, but companies can get their customers up and running if they put in the work to make sure it happens. We take this stuff very seriously at Foxtrot Breedables. We didn’t want to wait for it to be fixed by Second Life, we tried to find a way to get us fixed with as little downtime as possible, and I think we did that.
On behalf of Foxtrot Breedables, I want to apologize for the six-plus hours of downtime that our Second Life breeders experienced. I promise to keep such disruptions to a minimum, as long as it is in our power to fix them like today’s instance was.
- Why our breedables went down today in Second Life — but not in OpenSim - October 1, 2021