How to scale Second Life

Second Life had the potential to be the 3D AOL, to usher in a new age of immersive 3D environments.

It failed not because it was too complicated or difficult to use or tried to do too much at once.

After all, AOL also was a one-stop-shop: shopping, forums, email, instant messaging, personals, news, games — it wanted to be everything to all people. And AOL was NOT easy to use. You had to have a dial-up modem! You had to install software! And then you got a busy signal! You had to learn how to use a mouse! We forget how hard it was.

So I don’t think Second Life is too complex — just under capitalized. The Lindens could have gone out to strike deals to get it on everyone’s computer pre-installed. They could have run national ads everywhere. They could have made the subscriptions automatic and all-but-impossible to cancel. (I’m not saying they should have — I’m saying that AOL did these things.)

AOL spent $300 million on sending out those disks.  It covered the country with them. They were in every mailbox, at every checkout counter. You’d open up a new package of socks and — ta da — another AOL disk. AOL saturated the national consciousness and forced a critical proportion of the population to go online.

The question is — say Linden Lab pulled together $300 million, or a similarly large amount, and used it to launch a massive marketing campaign — would it be able to handle all the new users?

After all, the grid is already laggy, and regions have to be restarted all the time.

But that doesn’t mean that Second Life’s current size is cast in stone. It could change. It could evolve. It could become scalable.

Here are three possible options for how to do that.

Go to the cloud

(Image courtesy Tim Bates via Flickr.)

It could take a page out of Kitely’s playbook and move its asset server to the Amazon S3 cloud (super scalable, super fast, super cheap). And it could move its region servers to the Amazon EC2 computing cloud. Kitely says its regions can support 100 users and 100,000 prims — and they’re planning to raise both limits.

Kitely had to do a little fiddling with the OpenSim core code to make this work. But they’re just two guys working out of their homes. But given that their system works with the existing Second Life viewers — including all the official ones — it should be something that Linden Lab can do as well.

The end result won’t just be a more scalable system, but a potentially much less costly one, as well since regions don’t have to be running when they’re not being used.

 

Shard it

(Image courtesy V31S70 via Flickr.)

Linden Lab could “clone” its existing system. Massively multiplayer games already do this, setting up multiple parallel universes when the games get too large, and call it “sharding.”

Second Life could take that one step further and let folks hypergrid teleport between the different worlds. Linden Lab already tested hypergrid teleports between the Second Life grid and OpenSim, so there’s no reason why they can’t teleport between two different Second Life grids, as well.

Since the hypergrid system currently supports permissions, instant messaging, friends, and landmarks, Second Life could build a system of multiple worlds loosely linked to one another. Each separate grid handles its own logins, user inventories, and world map. If a user from a different grid teleports in, all inventory requests, messages, and other functions are simply forwarded to the grid of origin.

In the OpenSim universe hypergrid is usually thought of as an all-or-nothing kind of thing — either you allow anyone to teleport in and out, or you close down the grid and allow only residents to log in. But, in fact, hypergrid connectivity can be controlled — for example, two closed commercial grids could set up hypergrid links between one another, but not allow access to other, less secure grids.

Second Life could use the same mechanism to connect its grids together, allowing for infinite growth potential without a corresponding scalability problem on each individual asset server.

Use OpenSim

Yes, OpenSim isn’t yet fully compatible with Second Life. But its main areas of incompatibility — the social functionality provided by the Second Life website, and voice — Linden Lab can plug in. It also has a license for its physics engine, so it could extend that license to its OpenSim regions.

Linden Lab could offer standalone OpenSim grids — with an option of hypergrid connectivity back to the mainland — for estate owners, corporations, schools, communities, role playing groups, and other organizations.

Or it could simply use OpenSim to create low-rent residential worlds. Residential regions aren’t likely to need high-end physics — plus, it’s lower rent, so who’s going to complain? (Well, much, anyway.)

The latest versions of OpenSim are super stable (many users are reporting greater stability than Second Life when they run it on sufficient hardware), can hold over 1,000 avatars on a single region, hundreds of thousands of prims, and support mesh, media-on-a-prim, and all other recent bells and whistles. The latest version of OpenSim also supports bullet physics, which is supposed to be really good.

Second Life doesn’t cost $300 a region because of its servers or its software — it costs $300 a region because of access to its community. As a result, it could offer OpenSim regions at very little cost to Linden Lab itself, while charging customers a premium for access to the Second Life universe.

Related Posts

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

  • i am still on a year and a half old version of 0.7.1 dev (a dev branch from 2 summers ago) and it is way more stable than SL ever was for me. i had a good cross section of servers inSL with 19 sims – 9 were full sims, one was a stoopid expensive openspace and 9 were homesteads

    my OpenSim runs far far better – only one region has locked up ever and that was due to a looping script by DreamWalker. this week was the first time i had a clothing issue which was fixed by restarting the viewer, so i don’t think that was an OpenSim issue. i only restart the server about every two months for Window as updates, otherwise i never do anything with OpenSim except copy my automatic backups!

    my belief is that James (of SimHost) did a great job setting up the server. he installed Windows Server 2008 on it and then whatever else he needed. he tied it in to our domain DNS so the the box is at our domain. he built the box from the ground up and it is rock solid. he also has a very good and long standing relationship with the server centre

    OpenSim can run incredibly well!

    why would i ever go back to SL and their TOS? for our purposes, there is nothing compelling at all about going to SL – in fact it blocks some of our target audience

    Linden Lab is about making money and there is nothing wrong with that. Philip did SL to make money, lots of money, and he has an awesome home in one of the world’s most expensive real estate markets (he is next to Golden Gate Park in a 2700 sf home – i have an old post about his home right down to it’s location on google maps)

    LL will go down their Realms path or whatever half-assed Sims kind of plan and maybe that will work. as to commercial grids for people to be expressive and creative, InWorldz and Kitely are the ones who “get it”

    that culture is the vibe we all pick up on and will lead those groups to become more and more successful =)

    my canadian 2 cents before Hanukkah 

  • Try again, Maria.

    Virtual Environments such as SecondLife and OpenSim should *not* be striving to be the 3D AOL. They should be striving to be The 3D World Wide Web.

    For definitive proof of this, simply ask yourself a question:
    Of those walled gardens, which exist as they were originally designed?

    The answer is *none* – They were superseded by the world wide web.

    • Will —

      I totally agree with you. The benefit of AOL was that it heavily marketed the idea that people should go online — and opened up the way to the Web.

      After all, who’s going to market the WWW? It’s run by a non-profit, using opensource servers and open standards and, at the start, nobody was making money at it. So no big advertising budgets. But once folks got used to the idea of going online — they looked around and whoa — there’s a whole big web out there!

      I think SL could do the same — get folks used to the idea of going into an immersive environment and then they start looking around and whoa! There’s the hypergrid out there! Thousands of grids to explore! Maybe millions! (Soon!)

  • *laughing*  It already is the virtual world AOL with all the suckage that that implies.  Everyone that I know did not go online because of AOL.  They all despised AOL and Compuserve before it.  I agree with Will, the goal is virtual worlds and worldlets that are as easy to visit as clicking a link.  It is not for one vendor to rule them all or be the big winner.  The day of pushing a single solution successfully is thankfully long over.

    I never worked for LL but I understand that their architecture has some rather severe limitations on scaling, particularly on scaling how many avatars can be on a particular region, and also limitations on their main asset and user servers. All of us know of their limitations on messaging the outside world and of their seriously broken group chat implementation.  Judging from some of the problems of the past year it is pretty obvious that the architecture does not emphasize High Availability and Scaling all that much.   The federated approach might work sort of but I see no signs they are going there.  There asset management is not up to a much larger grid no matter.   Nor do I want silly LL restrictions to propagate.  Even something so simple as removing the 10 m restriction on prim dimensions is refused by LL.  And their TOS is simply infuriating.   So why pour this old wine into a bunch of new bottles?   People are leaving SL and/or exploring opensim for far more reasons than just the price difference. 

      Perhaps one vendor can provide much of the framework and tools these worlds exist within but even there well developed standards should allow competition on these services.   The world does not need more walled gardens.

    • Oh I agree there, that stupid 10m prim limit was crazy, as was their claim a region can hold up to 100 avatars when you would get lag starting with 20.
      The voice chat was the one great feature, but even that was buggy and would cut out or not work at times.

  • Interesting about the Cloud.  But the cloud can’t beat your own server farm except for on demand scalability.  It will run your just under $50 a month to run an EC2 small instance non-stop.  Yes, you can be smart and stop and start instances more on demand and multiplex them with region backups to some degree.  But it isn’t that cheap.  Especially not for a popular always up instance with lots of traffic (network cost).   I am reasonably technically competent but I think I would want RightScale or some other well versed EC2 support if I was doing this as a business – which of course adds a bit more cost.   I am curious what size instance Kitely gets this performance from.  I have run a diva 4 region standalone and also run it hooked into OSGrid.  Mostly it was fine but I didn’t attempt to stress it that hard. 

    • Hi Samantha,

      Kitely worlds are run on Large EC2 instances. Each Large EC2 instance hosts one or more worlds (each running in its own OpenSim instance) depending on how much load these worlds place on a server. Our system automatically allocates additional server resources to worlds with high concurrency so, when your world becomes popular, it may get its own Large EC2 instance all for itself.

      Small EC2 instances are not really suited for high concurrency due to inconsistent network performance on that class of virtual machine. Unless Amazon improves Small EC2 Instances’ performance reliability, I wouldn’t recommend them for anything but low usage scenarios.

      Our asset service is a proprietary cloud-based solution that utilizes S3 for long term storage and Apache web servers for HTTP asset transfers to viewers. There are quite a few such optimizations that a cloud-based solution can provide if you have sufficient development resources to create them.

      RightScale, and other generic scaling solutions, are not really suited for building a massively scalable OpenSim-based grid because the problems you encounter when trying to scale OpenSim are not easily solved by just starting additional servers.

      Providing just our current capabilities required us to write more than 150,000 lines of proprietary code (in addition to leveraging many open source projects). Considering that OpenSim includes just a bit more than 3 times that amount of code and was written by many more people over a longer period of time (see http://www.ohloh.net/p/opensimulator ) I wouldn’t downplay the effort that is required to build a solution that utilizes the cloud as more than just a way to get additional servers on demand. Building what Kitely built is definitely feasible but expect to need to re-engineer and work around a lot of OpenSim’s existing architecture.

      • Thanks for the very informative reply, Ilan.  I am glad you could find the resources to do that work.  I know that inWorldz and others also did some customization of some of the grid services such as assets.  I can see how taking advantages of the many cloud services that AWS offers would require a fair amount of re-architecture.   I have also noticed that running a region or a small megaregion on an EC2 small instance gave somewhat variable performance.  

  • Wayfinder posted:

    “Regarding the statement that Inworldz lost 19 sims since August… to be fair, I think a measly 19 sims out of a prior 867 is an extremely small percentage.”

    Wayfinder,  that was August to Sept, and Avination lost 80 regions at the same time, but you have to keep in mind two things- the loss of regions in IW v/s Avination are not the same bird at all!

    Why? because IW recycles their abandoned regions and RESELLS them, my two were up and active  for over three months after I abandoned them and said I was.
    Abandoning a region there does not mean the region is lost or goes away, it simply goes on a sales list to be resold to someone else, that means they dont really LOSE regions at all, because they arent deleted.
    The original OWNER may have walked away and abandoned the region, but that fact is not revealed on the region count/loss, for the simple matter that the region is still there and not deleted, for the same reason the “number of accounts” created is worthless because NONE are deleted either, but that’s true across all of the grids.

    On the other hand, Avination does NOT do that, they have a more sensible model where if the owner stops paying, the region goes into a hybernated backup archive that can be reinstated intact with reactivation up to 90 days I think.

    After that time it’s DELETED and gone, and that is the number that shows as a “region loss” simply because it HAS been deleted and purged from the system.
    This also is good because it protects the original owner’s region name that was used when it was installed, that name might be a registered business name or a brand name.

    “why you chose to leave Inworldz (and what you found more preferable elsewhere), ”

    A long story, but quite frankly Wayfinder, even if regions were FREE right now in IW I would not be interested, however, I will tell you some of the shortfalls with IW as well as the things I found more preferable in Avination, of course some things have changed I’m sure but here’s my experiences:

     in IW my regions constantly “cycled” the entire time they were there, it was like some other regions on the same server had a bunch of listener scripts running or who knows. My radar hud has color coding- red for bad lag, yellow for moderate and green for no lag, it would constantly change colors in a cycle lasting a couplke or three seconds- red, yellow, green, yellow, red, yellow and so on. I would look at the sim stats and they would be all over the map, 53 FPS, 30, 45, 12, 50, 3,  and so forth.
    As my regions had very few scripts of any kind and very few prims too- they were mostly wilderness scenery, this was nothing on my regions causing it.
    Installing raw terrain files was an excersize in futility when they wouldnt load more than about halfway and stop- backend issues.

    In Avination – identical regions, I was getting a near consistant 57 FPS, almost never below 53 FPS, the “cycling”/lag effect was not there, they simply ran extremely well, fast, borders were but a blip.

    installing raw terrain files went perfect and smoothly, I even uploaded several I had to look them over to see what they were, all loaded as expected.

    At the time we were not permitted to restart the regions in IW ourselves  though many of us did anyway during lag (restarting one region restarted ALL the others on the same server- geez what a setup that was!) to get a region restart you had to log into the forum and post a “request” and wait for it to be done on the back end…they usually got it done pretty quick, but what a pain, especially if you were doing an event.

    Region costs; $75 setup charge (avination has no such charge) $75 tier,  thats $150 up front to get the region in IW.

    Avination; $60 for the region, (now $40 special for 3 months) no other fees,  regions in IW often took  a couple of DAYS on upwards of a week to manually install after payment, Avination’s system installs them ready to log onto instantly upon payment.
    IW used a funky vendor thing on a region for the regions sales that was very difficult for people to use, I helped at least 2-3 people who couldn’t get it to work.
    Avination did it far more efficiently from their web site and the regions were installed instantly.

    IW has what Im reading a more or less customized, mostly proprietary version of opensim and physics engine, customized and proprietary are rarely good things except for those who own them. That’s similar to SL and to  AOL  years ago with their proprietary software and email no other email software can read.

     I believe the other grids all use free source open to all opensim with modifications shared with the others to benefit everyone, including local instances of opensim who benefit from the debugging and updates.

    Those are some of the reasons I abandoned two regions in IW (they were unsaleable to a private party even for $20 when the grid resells used regions for half price with a full month included) and moved to Avination and bought three regions.
    IW touts “45,000 prims” as a sales feature, but geez, since there is no way to save an OAR of your hard work- the more you pack in the less you can ever take back out if you decide to move,  or save a backup for, and none of my regions ever used more than 5,000 prims total anyway, so it was a moot point and I suspect a moot point for most. 15,000 prims is a lot of prims, and the more textures and objects you have the more performance suffers, 45,000 prims encourages jamming in a LOT of textures.

    With opensim you can have a mega-region with no borders and there IS no fixed limit on the number of prims (either 15,000 or 45,000)- it would depend on your machine and upload speed- 100,000 prims if you have the ram and machine’s specs.

    • Wayfinder Wishbringer

      That’s interesting… and adds further evidence to my belief that due to the nature of the software, individual experiences vary so widely as to be astounding.

      For example, I’ve heard some people say they can use SL… but can’t use OpenSim (“I crash all the time.”).  Other people say exactly the opposite, that they have no problem with OpenSim but SL crashes them regularly.  My experience is that Inworldz runs far more smoothly than either SL or OpenSim… which is one major reason I settled on Inworldz.  And yet a few of our group members report not being able to use Inworldz at all.  That strikes me as most strange… and seems to be a repeating issue, without pattern to give us a hint as why this is the case.

      From your post I determined you moved from Inworldz to Avination several months ago, during the height of Avination’s “popularity” phase.  It seems evident that something on that grid isn’t up to par, because while Inworldz  has been functioning and level for months now, Avination is taking a considerable nosedive.  I suspect something there is “going wrong” as Maria stated on another blog.  Apparently… people tried Avination and didn’t find it to meet their needs… despite the free setup and low initial sim fees. I’d be curious to find out why that population decline is happening. That would be important information for all OpenSim-style grids. It would help us all to know what to avoid, and might help Avination overcome their current slump

      Sometimes people don’t stop to think about incurred additional costs.  For example, I wrote an article on the Elf Clan blog listing one of the primary advantages of Inworldz over Second Life– that of Inworldz offering free uploads.   People don’t stop to think about how extremely expensive uploads are on SL, but to give a general idea:  uploading 10,000 textures would cost US$400 on Second Life… but is free on Inworldz.  If I remember correctly, when I checked out Avination some time ago they offered some 5,000 free uploads to established businesses (sorry new merchants and regular users), but thereafter charged. For a group the size of ours then, the upload fees would very quickly far, far exceed the setup fee on Inworldz.  But people usually don’t stop to think  about such things… especially newbs who have no idea how building and uploads work.  All they see is “free setup” vs “$75 setup”. 

      Inworldz is very well aware of the difference, and has opted to charge a setup fee and never charge for uploads.  They know in the long run, that’s a better deal for their customers.  That’s one thing I really like about Inworldz:  they put customer welfare first.

      If you experienced constant ups and downs and lag on your system and suspected server issues… my suggestion would have been to ask to transfer to another server (as we did, and the problems went away).   The very same thing you described has happened on Second Life ever since I’ve been a member there (over 7 years now), and the solution has always been to contact support and ask for transfer to a new server.  Moving to another grid would seem to me the hard way to solve such a problem.

      What I read between the lines above, is that you discovered reasons you were attracted to Avination and just decided to move.  That’s totally okay of course, but I don’t see that your core issues were problems with Inworldz itself, but rather external influences that brought about your decision.  Of course I can’t really judge that because as I stated, every user seems to have different experiences with different grids.  When I checked out Avination, my experiences were primarily negative.  My experiences with Inworldz were quite positive, so that’s where we settled.  Our group hasn’t regretted that decision in the year and a half we’ve been there.

      As for Inworldz being proprietary and not allowing entire sims to be downloaded to individual computers, you are correct.  I have a name for that:  security.   It’s designed to prevent copyrighted creations from being stolen and duplicated by others and is the only way to run an economy-oriented world securely.  So one has to decide which they prefer– a totally open and unsecure grid, or a closed and secure one.  Different strokes for different folks.  I certainly have no argument with either choice.

      Now while the claim is made that OpenSim can handle all the prims anyone wants to throw on a sim… that is in reality a moot point.  Because I’ve been assured by techs-in-the-know that while prims in excess of 45,000 is technically possible, the more one pushes the upper limits, the more lag sets in.  That’s why most grids limit their upper end at 30k to 45k.  It’s a sensible limit.

      So sure, it’s technically possible to do so, but sometimes just because something can be done doesn’t mean it should.   I heard of one OpenSim experiment where they bragged putting 140,000 prims on a sim with almost no lag… but those prims were untextured, unscripted, unspecial.   Try doing that with an average build (the kind which we create on Elf Clan lands)… and neither and Inworldz nor OpenSim region would be usable.  Everything would freeze, permanently.  Along with prims come scripts and textures (as well as simple graphics rezzing requirements)… and all that has to be considered in the equation.  To my knowledge, there’s no computer on the market that could handle a 100,000-primmed standard-build sim and survive the experience.  I do look forward to the day someone can prove that statement wrong. : )

      But this does point out something that regularly bothers me about OpenSim, OSgrid, Hypergrid, thisgrid, thatgrid and yes, even Second Life and Inworldz in some cases.  I often hear people making claims and bragging about things they can do… usually claiming their grid is better than yourgrid (my dad can whip yours!)… when those things in reality are pipe dreams,  supposed technical possibilities that have no realistic application.  A common claim I hear is “there’s nothing in the software itself that prevents such and such…”.  Well, there’s nothing in the software that counteracts the laws of physics and makes those features possible either.  They’re just “potential maybes” that are often hyped as if they actually exist somewhere outside of closed-circuit experimental laboratories.

      These potentials are things to look forward to in the future as computers and graphics cards become more powerful– and as the software itself becomes better coded (with all holes, bugs and flaws patched).  But at current time, such absurdist claims are just that– hype that seldom or never sees the light of day.  Someone claiming “You can have 150k prims on your sim if you want!” is a far cry from someone actually implementing that claim in full creative glory. : )

      So I don’t see the 45,000 prim limit of Inworldz all that limiting, nor the 256m size of sims.  As an example, on ElvenSong in Inworldz we have the Elf Clan home castle, DragonForge merchantile, a huge crystal castle (takes 1/4 sim, some 2000 prims in the build), a small 1/4 sim mini-Kingdom, the DwagonKeep Castle (huge thing, multitudinous rooms, fully furnished and scripted).   Above that we have Grunge City.  Above that we have Replicant City, one of Inworldz best-known attractions… a full science fiction city with multiple-genre museums, heavily scripted.  Above that we have the LostLove Cafe.   All in all, our sim is primmed and scripted from ground level to almost 4,000m in the air… and we still have almost 26,000 prims left. 

      So no… 45k prims isn’t really a limit on Inworldz.   On our 22 current sims, we haven’t even come close to hitting that limit. : )

      That said, I respect and appreciate your decision to move to Avination, and I’m glad you’re happy with your experiences there.  Why?  Because my general stance is there is no such thing as a “best grid”.  Every grid has its pros and cons, people it appeals to and those it doesn’t appeal to.  But in the end game, all these grids together are offering users an alternative to $295-a-month sims… which is a very good thing.   I believe that’s something you and I both agree upon. 😀

      • Wayfinder a couple of quick points, I have to run to work but will respond more in depth.

        1) $60 v/s $75 + $75 setup fee per month  v/s free uploads v/s 10 c dollars per upload;

        That $15 per MONTH, per region and that initial $75 differential will buy a heck of  a lot of uploads! your average region renter doesn’t run 22 regions as you do, and most people don’t upload 10,000 textures either.

        Over a short time that price differential adds up big time, sure I paid the uploads on Avination, but it cost me at best about $20 to completely upload everything, the first month’s  $60 v/s $75 with one region mean the first month I paid $5 more, the 2nd month I SAVED $15 in tier cost for that first region, the 2nd region came in a week and that saved $15 there, the next month and after I was saving $30/mo in Avination v/s the $150/mo inworldz was costing me.
        The 3rd region doesn’t count here because I had 2 in inworldz instead of 3, but had I had 3 there, the savings would have been $45/mo- one more region and it would have been like getting a free region over what I would have paid in inworldz.

        As I indicated, once you get set up, the savings is substantial over time, that $15/mo per region is $180 savings in a year. For a single region it’s not a lot, but if someone has 2, 3 4 or more regions thats a substantial amount of money differences that even if you upload more textures at a cost, the cost is like 4 cents or thereabouts- its peanuts- and it’s a one-time cost to store the texture there basically forever.

        That is not right about the uploads, avination offered upload credits for businesses and merchants on request and I remember it being a higher amount that you indicated, in any case a simple contact with their staff about individual circumstances and requesting a higher amount for a more valuable store or something is always the best thing.

        My leaving IW and abandoning 2 regions  had nothing to do with finding a better grid, in fact after my experiences with the grid founder elenia I went SEARCHING for an alternative to move to regardless of what the monthly cost was- it could have been higher or lower- my goal was getting away from that grid- PERIOD, even if it meant going back to SL and renting and consolidating to just one homestead region for $125-$150/mo.

        • Wayfinder Wishbringer

          I appreciate your honestly in mentioning that your reason for leaving Inworldz was due to a personal tiff with one of the Founders.  I can understand that.  Stuff happens.

          The statement you’d have returned to SL Homesteads rather than remain on Inworldz indicates to me your decision was one based on personal issues… without consideration for building power, cost, or the fact that Linden Lab has a long history of being a far more abusive entity.  That’s okay… we all have the right to make our personal decisions. 

          But that does take matters completely out of the realm that Avination offers advantage over Inworldz.  As I surmised in prior post, the reasons for exiting Inworldz were not because Avination was the better deal.  To be honest in return (just as an observation), what I’d seen in prior postings felt very strongly to me like a personal bias against Inworldz and clutching-at-straws to present the benefits of Avination.  I certainly understand that feeling and position;  I have a few issues with Linden Lab in much the same manner.  But I try to keep my statements regarding LL/SL at least based on accurate data and real experiences.  It doesn’t mean the comparative statments I’ve read regarding Inworldz are accurate… which is what I’ve hoped to point out in our mutually-friendly and respectful discussion.

          Sorry, the statement of $60 vs $75 is somewhat misleading, as Inworldz offers $60 sims as well. (I will grant that those regions allow only 30k prims and are not private islands). So the statement that Avination immediately saves members $15 a month, which provides them lots of money for uploads, while not totally unfactual– is misleading.

          I know our group is currently paying $50 a month for our sims on Inworldz (quantity group discounts), which is a $10 savings over Avination.  Of course, one has to be a large group to accomplish such fees.  But I hear some folks on Avination are paying a grandfathered $40 a sim (not sure how true that is).  What I appreciate about Inworldz is that their fees are solid and stable, available to everyone alike without marketing gimmicks or changing plans / special offers, and can be relied upon (I like reliability).  And every once in a while, if one is patient and bides their time, one can buy an Inworldz sim without setup fee (if they have a free setup special or someone is unloading a sim and it doesn’t require setup).  But those pricing games and claims can be played forever and has little to do with the overall value of comparitive grids.

          I would much more readily pay $75 a region on a grid like Inworldz that is doing its own dev work and focusing on customer needs, than pay $50 with a grid that’s not doing so (not referring directly to Avination, just making a point that price-of-region isn’t the only,  nor even primary issue in deciding on a grid.  Well, not for me.  For some people that may be the #1 factor.)

          Regarding Avinations “upload credits”… I’ve spoken to enough people to state that my presentation of such is accurate.  When I checked the grid myself, their initial upload allowance is 5k items, if you are an established, known merchant (or if they like you).  Once you reach that limit, whether or not you’re grated additional free uploads also depends on Avination owner’s mood at the time (I’ve been told that by more than one person with Avination experience… while in the Avination grid itself, and by reading their website).  So no, that information isn’t incorrect. 

          In comparison, Inworldz’ policy (and I quote from their website) is:  “All uploads are free.  We have no intentions of changing this in the future, as our goal is to allow creators to do what they do best:  create.”  It might also be recognized that the Inworldz Z$ (Izzy) is half the value of Avination’s credit… so that things on Inworldz tend to cost less in general due to the tendancy of that fact to lower overall prices.  It might also be mentioned that while Avination was established and widely advertised to be a merchant-focused system (ie, at first freebies weren’t even allowed)… Inworldz has always been a general-user focus, and far less concerned with economy overall.  From what I’ve seen, that has made Inworldz a relatively inexpensive environment when compared with other grids (grids that tend to charge similar merchandise fees to what we find on Second Life). 

          I really avoid comparing on grid to another (with the sole exception of comparing Inworldz to SL) because well, as I’ve said, I don’t believe there is any one “best grid”.  Everyone has different needs /desires / goals.  My only interest in all of this is simply to make sure the claims that are made about any grid are accurate.   I don’t like to dis anyone’s grid (not even SL, but sometimes LL does warrant it), don’t like to play the “my grid is better than yours” game, or try to influence people’s opinions about grids.  As the saying goes, “Just the facts, ma’am.”  : )

          Hope you have an enjoyable day at work.  Catch you later. : )

          • Yes “stuff happens, but as a paying  customer being talked down to and “dissed” on their public forum by the founder, I was no longer going to put up with the PROBLEMS either. I could accept the cycling lag, I could accept this bug or that  one, I could accept waiting a week to get a region after paying for it,  but when you add these all together PLUS the discourteous treatment from elenia, that’s when I re-assessed the whole situation and asked myself WHY the hell was I paying this woman $150 cash a month for 2 regions, plus spending money buying things there- totaling about $225-$250 per month averaged-  and be treated that way?

            I did not have time to detail that earlier, thus it was not just ONE thing and it was not JUST the fact of the discourteous treatment, it was a whole package of issues and the discourtesy on the personal level was the final straw as they say.

            Let’s face it, a paid for profit grid is like a cell phone company, you pay for a SERVICE each month, be it a region, a phone, your water/sewer, car insurance or anything else. You expect the company to treat you as a paying customer and provide the service you are paying premium dollars for, if they don’t meet that there’s usually several other alternatives happy to welcome you as a paying customer and provide the service or better service.

            Avination filled that bill in every aspect for me, my regions simply ran and performed for lack of a better word- flawlessly, the staff was very friendly, very helpful and on the ball. As a bonus the regions were cheaper saving me $30 a month which is $360 in a year, not small change.

            Actually when you consider Avination offers their regions for $40/mo for 3 mo then $60 afterwards, and no setup charge, the initial outlay for a new region  for 3 months there is $120 v/s $300 in inworldz, that’s a substantial difference! After the 3 months it would be $60 v/s $75, that’s still a considerable difference and Avination had many advantages I felt, but those were icing on the cake.

            Did Avination have problems? not for me personally that I saw or remember beyond the usual little things like logins disabled for a few minutes or an hour due to some network outage, and as I indicated before, I would still have 3 regions there today if running them from my dedicated home server for free was not an option.
            As it stands I have 10 regions.

            “Sorry, the statement of $60 vs $75 is somewhat misleading, as Inworldz offers $60 sims as well. ”

             But those are “MAINLAND” regions, which are also limited in other ways- terraforming is limited, prims, and other aspects. In some ways they are closer to SL’s cheaper “homestead” regions that were packed more to a shared server and allowed only 3,750 prims. 
            You cant really compare apples to oranges there, a private full control, set all your own rules and full terraforming and estate rights for a full $60 region in Avination is not the same as a limited $60 Mainland space in inworldz. If you go strictly by PRICE for a 256×256 area, well, $60 = $60 I guess 😉

            There you are with 22 regions $50 each per month  you are paying $1,100 a month EVERY month in inworldz, you only get the special rate BECAUSE you are paying for 22 regions, most people will never use that many, this is more of a business deal and I am certain that if someone went to Avination and asked about a 22 region discount they would get it.
            As they have their $40 region special price, I’m betting they would extend that permanently to someone who had 22 regions.

            For that kind of money however you could BUY a rack server or something and run 22 regions at no cost, or some of your members could each run 4 regions or something along that line, and donate all that  almost $13,000 wasted money to charity or a food bank or something.

            “As for Inworldz being proprietary and not allowing entire sims to be downloaded to individual computers, you are correct.  I have a name for that:  security.   It’s designed to prevent copyrighted creations from being stolen and duplicated by others and is the only way to run an economy-oriented world securely”

            We have a name for it; “walled garden” where you buy things and spend real cash on them but you are forever locked into the situation like SL where you can’t use anything out of the grid and cant save/backup or transfer items you PAID for.
            This is unlike buying a TV set and being able to take it with you when you move, or sell it, or give it away.
            People can copy anything they can see on their computer screen, not allowing OARS to be saved to a local drive does not prevent that any more than flickr.com putting transparent images over images to prevent downloading the images prevents people from simply taking a screen shot jpg.

            By “proprietary” I meant their customized SOFTWARE version of opensim.

            “Now while the claim is made that OpenSim can handle all the prims anyone wants to throw on a sim… that is in reality a moot point.  Because I’ve been assured by techs-in-the-know that while prims in excess of 45,000 is technically possible, the more one pushes the upper limits, the more lag sets in.  That’s why most grids limit their upper end at 30k to 45k.”

            That wasn’t my point, my point was IW uses “45K prims” as a sales point, but that sales point is moot when the region buyer only uses 5,000 prims. My other point was the prim limit is governed largely by your upload speed and ram (if running opensim) and if your ram is high and bandwidth good, you could have more prims if you needed them beyond 30k or 45k
            Also, a lot of people use free opensim to build, make clothes and test them out, avatars etc and develop things and then take them to SL where they sell them.

            On the uploads you said;

            “(I’ve been told that by more than one person with Avination experience… while in the Avination grid itself, and by reading their website).  So no, that information isn’t incorrect.  It’s quite factual”

            But this is legally called “hear-say”

            “I really try to avoid comparing one grid to another (with the sole exception of comparing Inworldz / OpenSim to SL) because well, as I’ve said, I don’t believe there is any one “best grid”.”

            Well, your posts now are sounding more and more like a sales promotion ad for IW than anything else to me, and I think we’ve pretty much covered it all.

            I’ve outlined my experience with these people as well as avination and others, and as far as I’m concerned people will just have to decide if they want to pay out $150  to get the first sim in IW, $40 to get one in Avination, $1,000 in SL, or run one from home for free and connect to a public grid like OSgrid for themselves.

            Take care.

      • I am not certain what I think about Inworldz.  I greatly miss hypergrid access.  I am not impressed with merchant friendliness when data on this site suggest that merchants top out at around $1500/month.  That is less than half a week of my day job earnings.  Not very impressive.  I do find my private region performance quite good.  It is very rare to see FPS dip below 40 and it is often in the 50s or higher.  I don’t like the lag relative to the opensim codebase.  I learned to take advantage of media on a prim in SL and I am not at all happy to not have it in inworldz.   To the good there are a ton of creative people on inworldz and a lot of very interesting things to acquire if one is into shopping more than looking at all the goodies and attempt to figure out how to build such things for oneself.  The shopping scene reminds me of SL back around 2007/2008.    

  • > It could take a page out of Kitely’s playbook and move its asset server
    to the Amazon S3 cloud (super scalable, super fast, super cheap). And it
    could move its region servers to the Amazon EC2 computing cloud.

    Using Amazon for Second Life would be much more expensive, not cheaper.

    It is expensive to run a 24/7 server on Amazon. Amazon is good to add and remove servers quickly. AWS based things like Kitely are good for lots of regions that are rarely used, but the opposite is true for Second Life which runs regions all the time.

    • Hi kripken 🙂

      You’re right that keeping servers active 24/7 on Amazon costs more than hosting dedicated servers in standard monthly plans but the great majority of the regions that SL runs see very little usage and could have been turned off when not in use to save money. A mixed approach may be preferable from a datacenter costs perspective but, if you are running a large enough proportion of your on-demand regions inside Amazon’s datacenter, there are various benefits to doing so for ones that run 24/7 as well.

    • For OpenSim regions, running them 24/7 on a dedicated instance of EC2 costs $72 a month — there are companies that will set it up for you. Yes, more expensive than your own servers, but a great short-term solution if you need lots and lots of servers for big events, or because of a sudden rush of customers. 

      It’s also a good option for little-used regions. There’s no reason to keep regions up and running 24/7 if they’re only used a few hours a month. Instead, simply activate them when folks log in or teleport in. Keeping empty regions up and running is a waste of money and energy. The only benefit you get is that you can see into the region when you’re standing on a region next door. And if that’s a requirement, you can also activate the region when folks visit any adjoining region.

      Some OpenSim hosting providers are experimenting with a hybrid system — using a private cloud with dedicated servers to run most regions, then throwing them up into the Amazon cloud for large events, or when demand suddenly exceeds the number of available servers and it takes time to bring new servers online.

  • Anonymous

    Totally agree with the move to cloud computing.  But not with the idea of having anything to do with hypergrid or opensim.  As an animation creator, I will never go near a hypergrid enabled world, which would allow anyone to steal my products simply by TPing to their own hosted sim then lifting the anims direct from their servers harddrive…

    • Wayfinder Wishbringer

      I have to sady agree with your statement on Hypergrid, Actingill, very much so. 

      This is a point I recently discussed with Maria and she stated Hypergrid was “secure”… but I woke up this morning with a fresh mind and not-at-4am-blogging, and realized that could not possibly be true.  HyperGrid can be likened to a busline that can get you from one place to another within a city… but isn’t going to protect you from the hoodlums when you get off in a bad neighborhood.

      I haven’t been keeping up well with the OpenSim side of things recently because I’ve had to focus on shutting down the last of our assets on Second Life (thank God) and shoring up our year-and-a-half pleasing existence on Inworldz (in other words, I been busy).   But from what I understand Hypergrid basically shunts people from one grid / region to another, and there is no telling who / what owns that grid / region.  Therefore Hypergrid’s suppoed “security” extends only so far as that region / grid owner can be trusted.  That means zero security.  Any claims to the contrary strikes me more as hopeful hype than reality.

      I always like to leave the way open for minimal difficulty in extracting foot from mouth, so if anyone can present evidence that Actingill and myself are totally mistaken in our understanding of this, I’m all ears.  But I think it is way past time we stop hearing claims that Hypergrid is as secure as private, closed-wall for-profit grids (a claim I’ve heard all too often).  I think that claim is made because security is the #1 claim of benefit of for-profit grids and I note a definite general bias against for-profit grids among some OpenSim users. 

      For the record, I *like* for-profit grids.  Saves me the extreme hassle of running my own servers… which is NOT as easy as OpenSim proponents constantly claim it to be… and is fraught with the constant possibility of sim data corruption.  I am all for OpenSim and OSgrid and even (if people wanna risk it) Hypergrid.  I applaud each and every honest and ethical effort at creating a user-controlled VR environment (I say honest and ethical, because there’ve been a few grid owners out there not so… and still are.  No surprise there.)  I have no respect for grids who try to snipe users from other grids, who misrepresent their capabilities / features, who lie about supposed lack of capabilities / features on other grids, who constantly play the “my grid is better than your grid” game.  That nonsense really isn’t necessary.  (If a grid owner doesn’t have the skillz to succeed without trying to snipe the users of other grids… they don’t need to be in business).

      Okay, rambling.  Your open and courageous statement about Hypergrid security pushed a button of “current pet peeves about the VR industry” and it all came just kinda flowing out.  It all boils down to:  why can’t everyone leave their bias at home and just play nice.  We can all brag about our preferred grids and invite folks to come visit us without viewing each other as… gasp… *competition*.   Imo the only real “competition” Opensim-based grids have is Linden Lab… which I would not trust or turn my back on for one second. We’ve been stabbed in the back by that company too many times already.  ; )

      But when it comes to OpenSim-based grids (including my personal favorite, Inworldz)… I applaud all of ’em. That means free, fee-based, private, public or top secret.  They all present alternatives to a $295 a month greed-based corporation that has destroyed more dreams and projects than most other companies I can think of.  OpenSim is a continuation of “the dream”… and I for one am all for the creativity and flexibiltiy presented by the whole lot. : )

      • If there were software proven agreements to how assets were handled between grids then it would be possible to hypergrid widely with little/no danger of theft except for the always possible dangers presented by a hacked client.   I am not sure this is technically possible but as a techie I am also not fully convinced it is impossible either.  

    • Actingill, Wayfinder —

      Again, hypergrid isn’t necessarily an “all or nothing” proposition. Yes, today, grids either have it all the way on or all the way off. But there’s also a middle ground — a grid could allow hypergrid teleports BUT ONLY TO OTHER TRUSTED GRIDS.

      So, for example, Linden Lab could set up hypergrid links between multiple Second Life grids. Any requests coming in from other grids would be ignored.

      Or, say, InWorldz could set up a hypergrid link to other grids with no-content-export policies.

    • We need more licensing models.  I am always happy to pay a premium for content that is either fully open or that I can use in any grid, especially my private machine regions, that I might find myself on.  The vast majority of customers are not dishonest but expect to wear that new frock or run that cool animation wherever they find themselves.  Nor is it imho that likely to run into the combination of a grid owner that cognizant of every cool thing that came onto ver grid or at least your cool thing, competent enough to find that cool thing’s bits, dishonest enough to grab them and repackage them for sale as ver own, and competent enough at business to actually be creditable competition. 

  • Wayfinder Wishbringer

    I would respectfully disagree with the concept that AOL is a good example of making the web more popular.  Why?  Because like Linden Lab, AOL generated about as many enemies as it did fans.  Techs especially hated how AOL took over the computer and was durn near impossible to uninstall.  People cursed AOL for continuing to charge them after they’d tried to cancel their account– repeatedly.  Imo… AOL is one really bad example of a way to run a company, and did as much to give the Web a bad name as encourage its use.  In truth, it seemed to me it was when AOL lost its stranglehold that the web really started booming. (Rather like how VR started booming once Opensim grids nixed SL’s iron-curtain monopoly).  I’d say this is just my opinion, but I notice a number of the comments below echo this observation.

    Regarding LL being undercapitalized… we began discussing this in another blog so to be truthful, I’m rather surprised to see the same claim being made here, again.   That being the case however, I’ll repeat my reply. ;D

    In an unofficial P&L analysis I did on the Elf Clan blogs, based on operative costs of similar companies I determined that Linden Lab’s monthly profit margin is likely in excess of US$5 million (that’s not revenue… that’s profit).  That is not a shabby amount (although of course, nowhere near the profit of systems such as World of Warcraft with its 11 million paid subscribers). 

    $5 million a month is $60 million a year.  That’s quite a bit of capitalization.  Now yes, while AOL spent some $300 million in advertising by sending out what people jokingly referred to as “coffee coasters” (ie, otherwise useless CDs that more often went in the trash as not)… that advertising budget severely damaged the company.  There’s only so much of one’s budget that should be put into advertising, and AOL vastly exceeded that limit.  Their failure in advertising savvy is legendary.  They’re no example for any company to follow, imo.

    Linden Lab would not have had to spend $300 million to accomplish affective advertising.  I came to SL in late October 2004 due to a 10 cent Google Adwords advertisement.  There are far more effective and less-costly ways to advertise a company than sending out millions of data discs.  So we have to believe the reason Linden Lab didn’t widely advertise was either because

    1) They were too cheap and profit-sucking to do so or…
    2) They believed their system could not handle the additional concurrency

    Myself, I believe both statements could very well be accurate.  I remember how SL functioned when it had 80,000 concurrent users on a regular basis.  It was awful.  I find it strange that Linden Lab has seemed to consistently work against adding additional concurrency to their product.  Second Life has been stagnant or declining for some three years now.   One has to wonder if that is possibly intentional.

    As several mention below, going “cloud” is likely not a solution LL would want to pursue financially.  Hard to say without knowing all the details of their operation.  Sharding?   Yes, that would seem a logical step:  split Second Life into multiple grids and allow hypergrid-like teleporting with full inventory access.  However, how does that overcome the very real issue of problematic LL asset servers?  That issue is something that Inworldz is currently tackling:  creating an asset server system that automatically grows as sims and customers are added, bringing zero additional system lag.  That’s what Linden Lab should have tackled long ago (as opposed to say, bouncing anatomy and a Viewer even a mother would hate).   Sharding in the form of having totally separate grids complete with separate inventory systems… well that might work and might bring more profits to the company, but– would customers support that concept?  I somehow seriously doubt it.  They’d want to be where their friends are– on the existing grid. 

    I’m going to withhold statement that “the latest versions of OpenSim are super stable”… except to say that I haven’t seen “super stable” anywhere, at any time, on any grid, in any instance.  My observation: your concept of “super stable” and mine is considerably different. ;D

    I’ll also have to disagree that Second Life charges $300 a sim for access to a community.  They charge $300 a sim because for years they were the only game in town, had a virtual monopoly, and people became so vested in it they now can’t afford to give it up (or don’t want to).  I think that is evident in that they have zero growth since Oct 2008.  I will very much agree that hundreds of thousands of people refuse to leave Second Life and move to OpenSim because of that community, friends and social access.  But the cost of sims is obviously not what the market will bear, and their social community not that alluring, or the grid would be growing.  OpenSim on the other hand, is growing… and those regions cost anywhere from $75 a month to free. 

    I think people are paying Linden Lab $295 a month for a few reasons:

    1) They have waaaay too much disposable cash
    2) They’re dumber than nails
    3) They’re big-time merchants or land barons who make a living from SL (in which case they’re probably paying $195… thus upsetting the inworld land economy)
    4) They have a group (which is indeed a community) and rent out lands to members to keep that group intact.  In that case though, it’s less a matter of people renting $300 sims… and more a case of people renting far less costly individual plots that happen to be on the same sim for group purposes.

    I think the time is going to come (as I imagine do most OpenSim fans)… that more and more SL members, groups and land owners will do the same thing Elf Clan did… wake up, declare they’ve had enough, and look for cheaper pastures.  I can say from experience, when that happens, SL community means zip… and they’ll cut the apron strings and take the community that they themselves have created, elsewhere than Second Life.

  • Unfortunately contrary to popular belief “the cloud” doesn’t magically solve scalability problems. The cloud solves infrastructure scalability issues, meaning it is easier to create and take down virtual servers than with real hardware. Besides for this and optional services (like key/value storage /S3), there isn’t much of a difference trying to scale software on standard hardware vs cloud hosting. 

    I think stating that clouds will solve all scalability problems is a great marketing tactic, and really great for trying to get VC interest, but in the end you are left with the same problems as someone scaling software on real (and virtualized) hardware as long as there are no limits to how much hardware is available to you.

    Second Life seems to have problems that could be solved by changing the way they’re thinking about scalability from the “sharding and caching” methodology to using more robust backend data storage systems that scale linearly and automatically. However, they may just have so much data that doing a live migration and not taking their services down would be a nightmare.

    At InWorldz we have worked almost exclusively on scalability and refuse to use any data components that do not scale, or that do not have a proven track record of scaling out. With our asset system already running on 4 cooperating servers, and a new inventory system based on a data storage technology that has proven linear scalability over 128 nodes, I think we’ll be able to prove this year that building your own “cloud” can be more cost effective for our usage scenarios than using someone else’s.

  • ” and I note a definite general bias against for-profit grids among some OpenSim users.  ”

    As an example, in IW I needed to outfit 2 regions with the forests I had in SL,, the only seller of trees was a guy who sold them in packs of usually 10 trees, well, I needed HUNDREDS of trees and none were copy, so that meant having to repeatedly buy packs of 10 trees, none of which were copy, and all of which were trash in the end because of the walled garden effect.

    When I set up in avination, again I had to buy ALL of those trees again, from the SAME seller as he had a store there too, forget that I had already paid for his trees, I had to go buy them all over again from him- what a joke.

    Never mind the fact that I PAID  a lot for all these items and could not longer use them for anything unless I continued to pay $150/mo for tiers.
    In SL I have dozens of expensive avatars, a number around $15 USD each not one can be backed up, saved to hard drive, or used elsewhere even though I PAID a  lot of money for these items-  they are totally worthless unless as planned- you continue in  SL.
    I can see not having things transferable, but when the items are purchased for PERSONAL use only, it should not matter where they are used by the original purchaser.
    That is why having zero economy and nothing for “sale” on an opensim, and everything free is good- you spend hundreds of hours designing and building a region and you can back it up safely and KEEP IT, and use it elsewhere, even locally.
    You cant do that in walled gardens, especially if you have bunches of regions and a group of people all building, not even second inventory will export the stuff and so you are either locked in forever to one grid, or you walk away and lose the time and money.

  • What is interesting when I look at Linden and Opensim is the reality that Linden is the biggest contributor (indirectly) to OpenSim at a significant level compared to anyone else. Without Linden you have no OpenSim and could seriously jeopardize the future of OpenSim if OpenSim could not keep reverse engineering out features that Linden spends massive amounts of money building.

    A few quick examples are.
    -The viewer (without this OpenSim would not even work)
    -Mesh (years of work from Linden)
    -Media support (again years of work from Linden)

    This all comes from Linden with Mesh and Media coming within the last 12 months. 

    I would argue that Opensim (using the AOL analogy) is really just a hybrid of AOL which relies completly on the mothership for its success. I think a better comparison of trying to find the ‘Apache’ version for 3d content I would go with WebGL.