Response: InWorldz at three

We recently celebrated our third birthday in InWorldz at a user conference in Las Vegas. The conference was fantastic, and was very well received. It also sparked an article “InWorldz at 3: Turning point or more of same?” by Maria Korolov. There were some very valid questions in the article, but unfortunately those questions got lost in the problems inherent in the article.

We believe those misrepresentations were due more to a misunderstanding of exactly what we’ve been doing on InWorldz with the code. While to the casual observer it may not seem “obvious,” there has been some very deep back end work done in terms of scalability and really only becomes apparent when the technology is pushed to its limits.

OpenSim, in and of itself is a fantastic tool for small private grids, home connections or standalones. Sim-on-a-Stick is one of those tools that would be considered marketing gold. It’s simple, relatively painless to install and gives immediate access. Inherent to OpenSim’s architecture is an upward scaling model, which makes it ideal for home users and private connections. However, once the software is deployed for larger commercial uses, the constraints of an upward scaling model become apparent in assets not able to become redundant, inventory not being fetched properly, and the load on the database resulting in slow or non existent retrieval times.

We’ve experienced all of this, and from that experience came some major work rewriting our foundation. I won’t go all geeky into the technology or how we’ve laid out our outward scaling model, for those who are so inclined to know more, you can read our TechBlog and David does an excellent job laying out what we’ve done.

When we started InWorldz, it certainly wasn’t something we ever envisioned being what it is today. We wanted a place to build on our own, to be able to capture that dream again. When we first installed OpenSim and saw the asset structure, red flags went up all over the place. Not from a business sense, but because I’m an asset hog personally! This was a deep concern because if we grew, what was going to happen with our assets from what we knew historically was the case of information storage in databases? This is not what databases were intended for. Therefore, it became our first major building block in our foundation after much debate between the three of us as Founders on the best solution.

Our users were then very happy over assets, but they couldn’t get 20 people on a 15,000 to 20,000-prim built up region without crashing for an event. That became our next goal, because our users are our most valued resource. In that work, a lot of memory leaks were cleaned up and memory handled much more efficiently. This also lead into region crossings as the last big crash problem. We were now able to handle 35-40 on a region, but it still wasn’t perfect. Over time we’d continue working on that with the new things found and cleaned up and fixed. This became our 2nd building block to the foundation.

Now our users had their assets secured, they had region crossings, could invite more avatars to a sim… they were not happy over always resetting their scripts such as animation overriders and vehicles on region crossings or server crossings. Phlox was then born and David wrote that. We now had our third major building block to the grid.

By the time Phlox was finished, we were getting reports of missing inventory, and yet they weren’t really missing. This gave us pause and we said, if we go straight to PhysX or other shinies, what will this do to our stability? People not getting inventory ranked quite higher than PhysX or Media on a Prim or mesh. Inventory was designed to scale outwards as well, and with that work complete, we now have 13 core servers all dedicated to simply holding assets, inventory, region information. All in redundant format, and with the ability that if 1 of those machines go down, no one in the grid will be affected while the machine is fixed.

It’s been said we’ve had “flat growth”, however, we encountered these issues as we grew. Imagine for a moment, if we had been out there advertising and marketing, discovering these things as we had a huge influx of growth. In point of fact, that did happen. In 2010, during our massive influx of regions, we saw our database come under extreme load and our systems tested to the max at about 305 people online. That was before Phlox, before Inventory, before the memory fixes. Our asset servers though… never lost a beat. They delivered every single asset requested and never stressed.

Maria asked us in private why we focused so hard on this core building of our foundation, and our answer is simple: because our users demanded it, because our users are our foundation, because our users are our investors. There is nothing more solid to us than our users. They speak the loudest, and to coin a phrase from Wayfinder, showstopper issues would make them leave in a heartbeat. Losing their inventory, their assets, crashing with events are all showstoppers to them and our service would not be worth paying a dime for.

The benefits of having done all this massive reworking of code? Well, they say a picture is worth a 1,000 words (many kudos to InWorldz Resident Alexina Proctor for taking this and other pictures in this set):

Mardi Gras 2012 at InWorldz. (Image courtesy Alexina Proctor.)

These pictures (of which there is a much larger collection here) were taken during the Mardi Gras fest on InWorldz. The stats? Fifty-eight avatars, 50,000 prims, more than 4,000 scripts. Crashes? None. One restart to increase avatar volume from the max 40. All regions are currently hard coded to a 50 avatar max limit now. Lag? Only viewer side. The server never even paused while people danced, rode on the floats, chatted and had a good time.

Mardi Gras 2012 at InWorldz. (Image courtesy Alexina Proctor.)

One question Maria asked us that was interesting: “When I ask people why they’re in InWorldz, they point to the community, to the people, the support they get from the Founders, the welcoming atmosphere. Do you have a vision here for a unique value proposition — what is it about the InWorldz community that people can get on your grid, but nowhere else?”

I sincerely think the answer is in her own question: “When I ask people why they’re in InWorldz, they point to the community, to the people, the support they get from the Founders, the welcoming atmosphere.” As we stated at our conference, the single biggest thing we realize is the absolute symbiosis of what we do. From the way we work with our partner at CARI.net, to working with our mentors, to our residents, it’s all a “trickle down” effect. Yes, we are known for our community. We have worked hard to build that community, and it goes through changes as do our residents. But without our residents, we have no company. Without our company we have no partnership. Without our partnership we’d not be able to focus so heavily on technology and hardware to drive the grid, and without the grid, we’d have no residents.

That in and of itself is our best feature. Our residents want to feel they are part of something bigger than themselves. They want to feel they make a difference, no matter how small it may be. They want to feel they are being heard, and that most of all (unlike other companies we all deal with day to day), they can speak up and help drive a vision of where this technology can go. We do not answer to a blind board of investors because our residents ARE our investors. Therefore, we can afford to listen to them, and go down the path they lay out for us with what they want. To date, this seems to be working. Will it always? Maria stated that a company has to anticipate what their customers want. I’ll state it a bit differently: a company has to listen to what their customers want, and anticipate changes their customers may not see.

So what do people get from InWorldz they don’t get elsewhere? We can sum that up in one word we hear daily: Home. That’s how they feel, that they are home. Does that mean it works for everyone? No. But for the market we’re aiming for at this moment, home means everything. Home means friends, family, community, but most of all, it’s comforting.

Providing that home for them, has presented us with many challenges along the way. Challenges we’ve met and dealt with. Challenges that forced us to abandon the scaling up architecture and scale out. Challenges that other grid owners will face as well. How they choose to deal with those challenges, is completely up to them, however, our doors are always open to talk to or advise within certain constraints.

I do wish to thank Maria for talking with us in private and allowing this to be posted. We sincerely feel she did not mean any harm in her article and had some very valid questions in there. Hopefully, this blog will answer a few of those, and explain our reasoning for proprietary code.

Last updated by at .

support@inworldz.com'

Beth Reischl

Beth Reischl is a founding partner of InWorldz LLC partner. She is also known as Elenia Llewellyn in-world.

  • http://twitter.com/Gaga_Gracious Gaga

    I have been in OSgrid for years. I have actually a very large inventory. I have never lost anything to speak of that I have noticed and rarely get a problem. My inventory loads pretty quickly most of the time and while it is true Opensim can be deployed at all levels it scales well for commercial use or there would be no commercial grids like Kitely using it. Inworldz is no longer Opensim of course so they may be different or better or worse. What matters at the end of the day is if the software works well enough and Opensim has proved to do that fine for most people.

    • http://twitter.com/EleniaLlewellyn Elenia

       I think the biggest thing most don’t see is the traffic Gaga. There’s almost a magic number when it comes to where you start to see the scale up model no longer work… between 275-300 concurrency (actual people, not bots). When that starts happening, you start seeing a massive load on the database. Then take maybe 10% of those people running at inventories larger than 80k items, and it starts to meltdown. This isn’t just true with InWorldz, it’s happened with other grids when they hit that amount of concurrency.

      I wouldn’t presume to speak for Kitely, as only Ilan knows what he’s reworked to work within the cloud structure. Only time will tell if those numbers hold true for them as well when their concurrency gets to that level.

      • http://twitter.com/Gaga_Gracious Gaga

        Well yes. Kitely has now separated their asset database to another cloud server if I remember rightly and this must be entirely possible for commercial grids running Opensim. For small standalones it will probably never be needed. The point is, Elenia Inworldz is doing a pretty fine job but I have always felt you put down Opensim as if to boost Inworldz code. You know, we have argued this point in the past and the recent comments in response to Maria carried a lot of that mud slinging against Opensim and OSgrid. I just think the ill will generated is a let down to the whole Metaverse and will only serve to put people off coming to join us. I have tried to make my own peace with Inworldz in the past year because I think we can be better than SL collectively.

        • http://twitter.com/EleniaLlewellyn Elenia

           And that’s why I pointed out it’s absolutely excellent for the smaller grids and standalones.

          The comments on the other article were more due to the discrepancies in the article, which is easy to understand if you’re not a regular resident they aren’t obvious. And yes, there was quite a rift formed which is why we contacted Maria privately. This blog is the culmination of those talks.

          The MetaVerse is a big wide open space, much like the internet it’s on, and has a lot of growth to go yet, and whole new markets and concepts to be discovered. And I don’t see any one single company being able to do it all, so there’s lots and lots of room for growth, both tech wise and creativity wise.

          • http://twitter.com/Gaga_Gracious Gaga

             I am glad Maria published your response Elenia and it really is a breath of fresh air over all after a distressing few days in which I felt an new rift opening up. I really didn’t want to see that happen when we have all been making serious progress on all fronts. SL is loosing hundreds of sims and the open Metaverse is growing.

            Dare I say it?

            We need to grow closer together too.

            Thank you Elenia *smiles*

          • http://twitter.com/EleniaLlewellyn Elenia

            Most welcome, we’re very glad she published it as well and gave us the opportunity to mend some bridges at the same time.

          • Eric Jones

            The best Grid Idea ever born. One of the best developers of viewers I know. The foundation is wonderful. and it will become the best system around in the future. But! Growing Pains
            one wonders after 6 months of IW looks at the people being run off of IW by the big owners and close friends of one of a founder sad to see the founder say these are my friends they will run the show you don’t like it your black listed. and so to all the people that been run off by the wicked witch of the west and her dear friends don’t waist your time bringing a product here as long as this one founder tries to let her close RL friends Ban there competition harass you and steel your designs. . The fact is no lag and a million prims and the finest grid ever designed has been spoiled by taking sides and hurting the people that pay the bills of inworldz grow up inworldz stop running people off because they are not your real life friend.
            Yes I know this sounds like a pissed off avi but tell you the truth go there and be the in-croud or be left behind.
            The fact is this has created another market that has to smile and hide from a micro managing Of the wicked witch . wow how does any body get there root prim attached to. only admin can do that.
            keep it up Inworldz there will be no more Luring them in.

          • http://www.facebook.com/eric.jones.921677 Eric Jones

            OMG I waited for 2 of my grad students to come inworldz we are putting together a media class there I started trying to explain what to do a over heated hall monitor. they call mentors ejects me while talking my class in you know what IW grow up you stop running the thinkers off.
            you may want to make a landing area for people that don’t want to put up with the Nazi mentality of your little police that ruined my class and ruined my work for the night shame on you mentor.

    • http://www.facebook.com/profile.php?id=1605956609 Ilan Tochner

      Hi Gaga,

      While OpenSim has improved a great deal since Inworldz forked their codebase from it, it still suffers from several architectural limitations, manly around how it handles assets and inventory. Inworldz chose to resolve those limitations using their proprietary asset and inventory servers and we at Kitely have developed our own proprietary solutions for those limitations as well.

      We’re already using our own cloud-based solution for asset handling which automatically scales with load. Part of the work we did for the asset handling was in preparation for moving to cloud-based handling of inventory as well, so we can complete development within a few weeks if the need arises. For the time being, inventory handling is not creating a lot of load on our grid servers so we prefer to focus on other user-facing features first. We do, however, plan to replace the standard OpenSim inventory service with a more scalable system before it starts effecting performance.

      To summarize, OpenSim, in its current standard configuration, is great for many things but still needs work on improving some key architectural components before it can scale while maintaining consistently good performance. Kitely has done work to resolve those issues and other OpenSim-based grids with competent developers can do the same.

      • http://twitter.com/Gaga_Gracious Gaga

         Hi Ilan

        Yes, I said as much in my reply to Elenia so thank you for expanding on it and clarifying.

        BTW, I have now joined Kitely and visited using Twitter.

        I will be blogging soon about Kitely and Inworldz

        Gaga

        • http://www.facebook.com/profile.php?id=1605956609 Ilan Tochner

          Great, I’m looking forward to reading your blog posts :-)

      • http://twitter.com/TranquillityIW Tranquillity Dexler

        Thank you for sharing your experience Ilan. I think this helps to confirm what we’ve done was well as work that will need to be done in the future. As always, it’s been a pleasure sharing experiences and statistics with you. I hope the collaboration and information sharing can continue into the future.

        Sincerely,
        - David

        • http://www.facebook.com/profile.php?id=1605956609 Ilan Tochner

          Thank you Tranquillity,

          I agree, our collaboration can help benefit the development of Inworldz, Kitely and the metaverse community at large.

          I look forward to working together and hope that one day people will be able to easily teleport between Inworldz and Kitely using a safe and content-rights-respecting method.

  • http://www.facebook.com/people/Linda-Kellie/100002704826887 Linda Kellie

    Great post Elenia. It clears up so much bogus information that people tend to throw around. I do wish that people would realize when writing about InWorldz that they are a company and should be treated the same as any other company and find out the facts before posting and doing damage to a business. 
    About the inventory issue: I do wish that OpenSim inventory worked as well as InWorldz. I rarely lose anything in my inventory in OSGrid but I do get things coming back to me all the time and I have things doubled and tripled. Even no copy things that I have picked up along the way will end up being in my inventory a few times in a few different places. What is even worse is that people who pick up my creations and then delete or return them (not sure which) it will end up back in my inventory showing them as the owner. I spend hours and hours just sorting and having to look at the properties of every item to see if I am the owner of the stuff in my own inventory. And with an inventory of about 17 thousand items now it’s frustrating. I’m loving OSGrid and hypergrid. But more and more I am craving the structure and the stability on InWorldz.Not missing SL even a little bit though. :) 

    • Jigs

       You get inventory back because in OSgrid you have to go to the web site, log in and empty the trash THERE, emptying trash in-world in your inventory window does not work and will not ever function that way.

  • Justin Clark-Casey

    A few points.

    * I don’t think that technical like-for-like comparisons between ‘closed’ grids and ‘open’ grids like OSGrid are particularly fair.  Whilst ‘closed’ grids control all their simulators, ‘open’ grids like OSGrid allow third parties to hook up their own systems. 

    This means that the ‘open’ grid experience is at the mercy of people running OpenSimulator on very different hardware (some of which may be underpowered) and on very different networks, which will affect simulator to grid service operations (e.g. inventory) as well as simulator to simulator operations (e.g. teleporting).

    As an aside, ‘open’ and ‘closed’ are not meant as pejorative terms – they are just the ones that came to mind when I first wrote about this [1].  As I also wrote in the reference, I think ‘open’ and ‘closed’ grids fulfill different needs and both are equally valid.

    * I think of running a medium-sized grid akin to running a pretty big website at this point.  There is a lot that you have to do behind the scenes to keep things running smoothly.  Big websites spend a lot of time on their architecture.

    To be honest, the default services that come bundled with OpenSimulator are probably never going to let you run a mega-huge grid (though they will get better over time).  And that was never their intention – they exist as enough to get you going and quite happily run smaller-scale installations.

    Rather the scalability is left to dedicated third-party projects.  For instance, SRAS (Simple Ruby Asset Service) [2] is an open-source Ruby on Rails replacement ROBUST asset service that can plug straight into core OpenSimulator and provide asset deduplication, compression, etc. (it’s used by OSGrid, amongst others).  Since SRAS runs on webservers such as Apache rather than on the possibly unoptimized embedded C# webserver used by OpenSimulator, it performs much better.  However, it is arguably more complex to get up and running and to maintain so it’s not something that can be an OpenSimulator default.

    This is exactly the kind of thing that happens on large websites (Facebook, Twitter, etc.).  They spend a lot of time on performance and plumb in a large number of different projects.  And from all that I know, a large number of these projects are open-source.

    This isn’t to say that there aren’t improvements that need to be made to core OpenSimulator to make it work better with services.  But these will happen over time as people come to grips with the problem in many different contexts and contribute knowledge and code, whether their other components are open-source or proprietary.

    * Which brings me on to a more philosophical point.  I believe that companies work best in an ecosystem if they share a lot of the base technology as open-source.  That’s not to say I have anything at all against proprietary software – indeed the website examples I quoted have a good proportion of that as their special source.  But at the same time they make use of and contribute a good proportion of their work as open-source.

    My belief is that this is essential to build a healthy virtual world/environment ecosystem (as opposed to a single game system).  I don’t think any single company will be dominant in this space in the long term – the risks are high and the network effects are potentially large (e.g. being able to use the same viewer across multiple systems, hypergrid or similar systems).

    So in the long term I think using and helping to improve open-source VE/VW projects, whether server or viewer-side reduces a project’s operating costs, keeps them compatible with other systems (if such is desired) and allows them to spend time innovating in their own specific area (whether technical or in areas such as community building).  But I accept that this is a personal point of view with many nuances and counter-arguments.

    * Lastly, regardless of whether Maria’s report [3] was accurate or not, I do want to say that I was disappointed by many of the comments making comparisons between InWorldz code and OpenSimulator.  OpenSimulator 0.6.5, for instance, was released in May 2009 – almost 3 years ago.  Since then there has been an enormous amount of improvement in core OpenSimulator in features, bug fixes and code refactoring.

    I’m quite happy to see informed and evidence-based criticisms of current OpenSimulator code and performance – after all, that’s the only way we get better.  But comments that just assert that the code is poorly performing or a conceptual mess without giving any evidence to back those statements up are, to my mind, rather pointless.

    [1] http://justincc.org/blog/2008/04/24/open-grids-and-distributed-asset-systems/
    [2] https://github.com/coyled/sras

    • Jim Tarber

       I completely agree that comparisons between open and closed grids is not very fair or appropriate.  But not really for technical reasons.  They serve different purposes, and they serve different markets.  It would be completely inappropriate for me as someone interested in one purpose and market to suggest that the other type of grid completely change what it’s doing, because it’s not the purpose and market that I’m interested in.  And to abandon all the work done so far.  But that’s open grids vs. closed grids.

      Similarly, when discussing which server codebase to use for a large concurrency grid (of either kind), the trade-offs are very different than those of smaller, run-at-home grids, or even smaller grids that run in the cloud and server less than 250 concurrent users.  Those grids simply don’t need all the highly-scalable components that grids like InWorldz and Kitely have developed.  MySQL is fine for small- to mid-sized concurrency.  Passing XML around all over the place, even between components within a single machine, is fine as well.  It creates problems under higher load, but unless the open grids take off dramatically, the priorities for those open grids lie elsewhere.  It makes sense that the spread of items and users across grids is one of the higher priorities there.  And other user-requested features can sit closer to the top as well.

      However on a closed grid where there is an economy and significant investment of content/IP, this is not the case.  Reliability, scalability, redundancy, content protection, land access controls, security of the servers, etc… these are all the highest priorities.

      On open grids, the content is portable. OAR files are common, and people know how to load them and save them on their own home-based regions.  It’s all about sharing and extending access.  It’s the complete opposite.  On closed grids, it’s all about knowing that the grid administrator (someone other than you) has protected your investment of time and money through backups, redundant hardware, reliability and scalability.  And that the grid is there for the long term.

      There are very different goals here. They serve different purposes and markets.  It’s okay to like both philosophies.  In the future I see them working together.

      • Fred Leftwich

        “In the future I see them working together.”

        I hope so Jim.  It is totally understandable why you have a vested interest in the proprietary technology of Inworldz.  For you to be professional and committed to your employer there is really no other alternative.  To be honest, I logged in recently and noticed a *significant* performance difference compared to normal usage on OpenSim (well OSgrid).  There is good work being done for sure.

        I can’t help but think of OS/2 Warp.  It was amazing technology at the time, better than Microsoft’s.  Certainly the product of a lot of talented developers at IBM.  But where is it now? Yes, some vestige remains, but it was ultimately relegated to a footnote in the history of OSes.

        But I think you are a forward thinker.  I get the feeling you have a perspective to offer Inworldz that may allow them to integrate (maybe delegate) key developments into the future of the metaverse that will allow Inworldz to realize profit while navigating in harmony with the general trend instead of swimming upstream against it.

  • Justin Clark-Casey

    By the way Linda, I believe the bug where you end up getting copies of items that other people pick up is fixed in OpenSimulator 0.7.3 and later.  Sims on osgrid might still be running older code as a forced upgrade hasn’t been required for a long time. 

    • http://www.facebook.com/people/Linda-Kellie/100002704826887 Linda Kellie

      All of my sims are 7.3 but I guess maybe someone is deleting/returning them from a sim running an older version since OSGrid doesn’t control which version people choose to use on their own hosted sims. 
      It’s not really an issue for me now that I am changing course but thanks for addressing it. 

  • http://www.facebook.com/fhendrickson2 Fred Hendrickson

    I have been using opensimulator as a way to understand how a virtual world grid architecture is working as I develop my own grid.  Ive also watched how both SecondLife and Inworldz have been operating.  I would say this however,  in my studying and watching I would be more apt to agree with Justin.  Inworldz and Opensimulator should not be compared as they both have two separate goals. However one thing I did notice is the confusion about what Opensimulator version Inworldz is running could easily be solved.  Inworldz in my opinion is not running the stated version of 0.6.5.  I can say this because I remember getting quite annoyed when I couldn’t get that version to work even after I followed all the directions and then finally got it to work after taking a break.  I would say that Inworldz is trying to implement its own grid architecture while using opensimulator as its base.  Therefore what version of opensimulator wouldn’t and shouldn’t matter as its a closed grid.  All the founders need to do is fix the default library name in the inventory to not show Opensim Library and that will solve that confusion.  Their simulators servers show Inworldz 0.7.0 as the server version which does seem correct and they are making improvements.  Keep up the good work.

  • Jigs

    I see the other threaded article is locked now, some were posting these humongously insanely long comments there that scrolled down 3 screens worth of solid wall text, maybe a post comment length maximum would be a good idea to have. One post there is 142 lines long, 3,244 words and almost 20,000 chars-  no one is going to sit there and read a web site version of the monumental book  “War and peace”!