We recently celebrated our third birthday in InWorldz at a user conference in Las Vegas. The conference was fantastic, and was very well received. It also sparked an article “InWorldz at 3: Turning point or more of same?” by Maria Korolov. There were some very valid questions in the article, but unfortunately those questions got lost in the problems inherent in the article.
We believe those misrepresentations were due more to a misunderstanding of exactly what we’ve been doing on InWorldz with the code. While to the casual observer it may not seem “obvious,” there has been some very deep back end work done in terms of scalability and really only becomes apparent when the technology is pushed to its limits.
OpenSim, in and of itself is a fantastic tool for small private grids, home connections or standalones. Sim-on-a-Stick¬†is one of those tools that would be considered marketing gold. It’s simple, relatively painless to install and gives immediate access. Inherent to OpenSim’s architecture is an upward scaling model, which makes it ideal for home users and private connections. However, once the software is deployed for larger commercial uses, the constraints of an upward scaling model become apparent in assets not able to become redundant, inventory not being fetched properly, and the load on the database resulting in slow or non existent retrieval times.
We’ve experienced all of this, and from that experience came some major work rewriting our foundation. I won’t go all geeky into the technology or how we’ve laid out our outward scaling model, for those who are so inclined to know more, you can read our TechBlog¬†and David does an excellent job laying out what we’ve done.
When we started InWorldz, it certainly wasn’t something we ever envisioned being what it is today. We wanted a place to build on our own, to be able to capture that dream again. When we first installed OpenSim and saw the asset structure, red flags went up all over the place. Not from a business sense, but because I’m an asset hog personally! This was a deep concern because if we grew, what was going to happen with our assets from what we knew historically was the case of information storage in databases? This is not what databases were intended for. Therefore, it became our first major building block in our foundation after much debate between the three of us as Founders on the best solution.
Our users were then very happy over assets, but they couldn’t get 20 people on a 15,000 to 20,000-prim built up region without crashing for an event. That became our next goal, because our users are our most valued resource. In that work, a lot of memory leaks were cleaned up and memory handled much more efficiently. This also lead into region crossings as the last big crash problem. We were now able to handle 35-40 on a region, but it still wasn’t perfect. Over time we’d continue working on that with the new things found and cleaned up and fixed. This became our 2nd building block to the foundation.
Now our users had their assets secured, they had region crossings, could invite more avatars to a sim… they were not happy over always resetting their scripts such as animation overriders and vehicles on region crossings or server crossings. Phlox was then born and David wrote that. We now had our third major building block to the grid.
By the time Phlox was finished, we were getting reports of missing inventory, and yet they weren’t really missing. This gave us pause and we said, if we go straight to PhysX or other shinies, what will this do to our stability? People not getting inventory ranked quite higher than PhysX or Media on a Prim or mesh. Inventory was designed to scale outwards as well, and with that work complete, we now have 13 core servers all dedicated to simply holding assets, inventory, region information. All in redundant format, and with the ability that if 1 of those machines go down, no one in the grid will be affected while the machine is fixed.
It’s been said we’ve had “flat growth”, however, we encountered these issues as we grew. Imagine for a moment, if we had been out there advertising and marketing, discovering these things as we had a huge influx of growth. In point of fact, that did happen. In 2010, during our massive influx of regions, we saw our database come under extreme load and our systems tested to the max at about 305 people online. That was before Phlox, before Inventory, before the memory fixes. Our asset servers though… never lost a beat. They delivered every single asset requested and never stressed.
Maria asked us in private why we focused so hard on this core building of our foundation, and our answer is simple: because our users demanded it, because our users are our foundation, because our users are our investors. There is nothing more solid to us than our users. They speak the loudest, and to coin a phrase from Wayfinder, showstopper issues would make them leave in a heartbeat. Losing their inventory, their assets, crashing with events are all showstoppers to them and our service would not be worth paying a dime for.
The benefits of having done all this massive reworking of code? Well, they say a picture is worth a 1,000 words (many kudos to InWorldz Resident Alexina Proctor for taking this and other pictures in this set):
These pictures (of which there is a much larger collection here) were taken during the Mardi Gras fest on InWorldz. The stats? Fifty-eight avatars, 50,000 prims, more than 4,000 scripts. Crashes? None. One restart to increase avatar volume from the max 40. All regions are currently hard coded to a 50 avatar max limit now. Lag? Only viewer side. The server never even paused while people danced, rode on the floats, chatted and had a good time.
One question Maria asked us that was interesting:¬†“When I ask people why they’re in InWorldz, they point to the community, to the people, the support they get from the Founders, the welcoming atmosphere.¬†Do you have a vision here for a unique value proposition — what is it about the InWorldz community that people can get on your grid, but nowhere else?”
I sincerely think the answer is in her own question: “When I ask people why they’re in InWorldz, they point to the community, to the people, the support they get from the Founders, the welcoming atmosphere.” As we stated at our conference, the single biggest thing we realize is the absolute symbiosis of what we do. From the way we work with our partner at CARI.net, to working with our mentors, to our residents, it’s all a “trickle down” effect. Yes, we are known for our community. We have worked hard to build that community, and it goes through changes as do our residents. But without our residents, we have no company. Without our company we have no partnership. Without our partnership we’d not be able to focus so heavily on technology and hardware to drive the grid, and without the grid, we’d have no residents.
That in and of itself is our best feature. Our residents want to feel they are part of something bigger than themselves. They want to feel they make a difference, no matter how small it may be. They want to feel they are being heard, and that most of all (unlike other companies we all deal with day to day), they can speak up and help drive a vision of where this technology can go. We do not answer to a blind board of investors because our residents ARE our investors. Therefore, we can afford to listen to them, and go down the path they lay out for us with what they want. To date, this seems to be working. Will it always? Maria stated that a company has to anticipate what their customers want. I’ll state it a bit differently: a company has to listen to what their customers want, and anticipate changes their customers may not see.
So what do people get from InWorldz they don’t get elsewhere? We can sum that up in one word we hear daily: Home. That’s how they feel, that they are home. Does that mean it works for everyone? No. But for the market we’re aiming for at this moment, home means everything. Home means friends, family, community, but most of all, it’s comforting.
Providing that home for them, has presented us with many challenges along the way. Challenges we’ve met and dealt with. Challenges that forced us to abandon the scaling up architecture and scale out. Challenges that other grid owners will face as well. How they choose to deal with those challenges, is completely up to them, however, our doors are always open to talk to or advise within certain constraints.
I do wish to thank Maria for talking with us in private and allowing this to be posted. We sincerely feel she did not mean any harm in her article and had some very valid questions in there. Hopefully, this blog will answer a few of those, and explain our reasoning for proprietary code.