Kitely moves assets to Amazon cloud

On-demand hosting provider Kitely Ltd. has eliminated a major OpenSim scalability obstacle by moving its asset database to the Amazon storage cloud.

“This is the most significant change we’ve made to our system since our beta started,” Kitely CEO Ilan Tochner told Hypergrid Business.

Because of the upgrade, Kitely was down for eight hours on Monday while data was migrated from the old database to the Amazon cloud, Amazon S3.

Previously, Kitely was using Amazon’s computing cloud for the virtual servers running individual OpenSim regions — but all the assets, such as avatar inventories and in-world content, was stored in the standard OpenSim database. Since this one database had to handle all the content for all the avatars and regions, it was a significant bottleneck.

“The standard OpenSim asset server just doesn’t scale well,” Tochner.

Now, the database is mostly eliminated, and individual assets are stored in the Amazon storage cloud — which can handle an almost unlimited number of simultaneous access requests.

According to Oren Hurvitz, Kitely’s vice president for research and development, the new system, the Comprehensive Assets System (Cassy) will be used to store all the textures, sounds, animations and other content used in OpenSim.

“Cassy enables Kitely to have faster world startup times, less server lag and provide faster viewer rezz times than grids based on standard OpenSim,” Hurvitz said in a blog post.

Cassy also detects duplicate assets, he said. Kitely users frequently upload content from popular OpenSim content sites such as OpenSim Creations and — Cassy checks to see whether the content has already been uploaded before, and stores just one copy.

The Linda Kellie region is a popular upload for Kitely.

“We also improved how viewers download textures,” said Hurvitz. “Textures are the largest and most common type of asset, and viewers spend a lot of time downloading them from OpenSim… To lighten the load, we now serve textures from web servers running Apache. When a viewer requests a texture OpenSim redirects it to Apache, where it can download the texture. Apache is much more efficient than OpenSim at serving files so this makes the server run more smoothly, and viewers can display worlds faster.”

Cassy is also used to speed up script loading, he said.

“This means that in most cases script-heavy worlds that would take minutes for all the scripts they include to start running in standard OpenSim will take seconds to start running in Kitely,” he said. “Scripts still need to be compiled when a newly imported OAR file is first started, which may result in slower startup times, but after the scripts have been compiled worlds should now become fully functional much faster than they did previously.”

Kitely regions are stored away when not in use, and loaded up only when needed. A major goal for Kitely is to reduce region load times to the point where users can teleport in and not notice any delays.

Other grids don’t have the same problem since regions are always up and running. But if Kitely is able to reduce region startup times far enough, other grids may also begin looking at on-demand regions.

The old OpenSim asset database isn’t completely gone, however, said Tochner.

“Assets themselves are no longer stored in the database but other things are, such as some metadata about assets,” he said. “We could have moved that to S3 as well but it wasn’t worth the current effort to do so. Metadata is tiny compared to the assets themselves so even a single server can handle orders of magnitude more load when it just has to deal with metadata.”

Kitely also used the upgrade as an opportunity to roll out a couple of other features, including a fix for terrains showing up striped, enabling physics for meshes to eliminate invisible boundaries around mesh objects, and increasing the size limit for uploaded OAR region files from 250 MB to 1 GB.

Related Posts'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China. Follow me on Twitter @MariaKorolov.

12 Responses

  1. soror nishi says:

    This all sounds very interesting. I have a project in mind … shame there aren’t more hours in the day…

    • Ilan Tochner says:

      Hi Soror,

      Using Kitely is currently free so there is no better time to start your new project than the present  🙂

      Even a few minutes each day may be enough to create something of value… Worst case you enjoy the process of building on your own sim that supports up to 100,000 prims and 100 concurrent avatars without needing to spend a dime on setup or maintenance fees.

  2. Ener Hax says:

    very nice!!!  congrats Oren and Ilan! =)

  3. Ilan Tochner says:

    Thanks Ener,

    We’re settings things up so we’ll be able to support future user-base
    growth once we remove some of the current barriers to adoption, e.g. the
    current lack of alternatives to using Facebook for accessing our

  4. Ilan,

    You really really really (i’d put another really) need to test your simulation with 100 real avatars on real loaded sims before you start promoting it. Just take my word for it.

    • Ilan Tochner says:

      Hi Tranquillity,

      A sim running on Kitely with 100 avatars will eventually (it’s automatic but not instantaneous) get a dedicated Large EC2 instance to run on. That has the processing power of about a Class 5 Second Life server but with about twice the memory (7.5 GB on the EC2 server compared to the 4 GB on the Class 5 SL server). It also has dedicated access to 1 Gigabit network connection to the internet. This is the same level of hardware that other well reputed OpenSim hosting providers usually run 4 separate sims on, each such sim marketed as supporting 45-80 concurrent avatars.

      Our new asset server system (S3, Apache, etc.) removes a lot of system load that usually kills sims as concurrency increases. Our new architecture helps scripts start a lot faster and significantly reduces the load on the sim caused by needing to transfer assets to people’s viewers.   

      Scripts and inventory support is still at standard OpenSim level (thou we do start scripts and transfer inventory much faster now) so those can definitely effect sim performance as avatar concurrency grows. That said, our system provides the sim with more CPU, memory and bandwidth to work with as concurrency raises to help compensate for that. The sim should get an entire Large EC2 instance dedicated to it way before it reaches 100 concurrent avatars.

      There is still a lot of work to be done on optimizing and stabilizing OpenSim (especially in regards to the script engine and how inventory is handled) but we feel quite confident that our system should be able to handle 100 real avatars on a sim hosting an asset rich world.

      • You sure? You realize some of the very intensive processes with that many avatars aren’t paralleized and thus will bottleneck well before maxing out CPU cores? Is your process 64 bit, or 32 bit?

        I’ve done some of these experiments already with real avatars, which is why I’m curious about if you’ve tried it in a real life scenario yet. We run Xeons just like amazon, but all those cores don’t mean much of the software isn’t using them efficiently.

        I would take a real build, at least 15k prim+.. I might suggest even more if you’re advertising 100k prims now. And put as many real avatars on it as you can.

        • Ilan Tochner says:

          We’re using 64bit across the board. How IO load is handled has been significantly changed with our new asset system, asset-related services that were centralized have been distributed (S3, HTTP asset transfer was removed from OpenSim, etc.), several asset-related single thread processes in OpenSim were made multithreaded, and several things were done to improve script handling (thou the script engine remains the same as in standard OpenSim).

          I’m aware that standard OpenSim scripts and inventory handling (which we still use) can cause IO, memory and CPU load but those should be handled at least partially by having access to more bandwidth, memory and CPU resources (which our system handles automatically). Can you please enumerate the single threaded intensive processes you beleive will form a bottleneck at high concurrency?

          • Asset requests never had anything to do with a sim staying up or being acceptable under high load for us. In fact if the assets are getting to the sim faster now, and there is processing involved (think UDP textures, which most 1.x viewer people still use unfortunately), you will see an increase in temporary memory demand and generation 2 GC collects (CPU intensive). However, if your asset client was processing XML or Base64 encoded data or something and it isn’t now you’ll probably see a large benefit (i dont know how the current opensim transfers assets back and forth anymore). 

            I assure you our requests are very efficient and low latency already and the only thing it helps with is rez times, script load/save times, etc. It can actually end up creating a larger burden on the sim simply due to the fact that more assets are available sooner to be processed in parallel where applicable.Inventory loading can be a large issue, but there is something bigger that you’ll have to worry about at that level of activity. Think about the multiplying effect of that many avatars and the systems that impacts when people move, or objects change, etc. You really need to profile. I’ve already posted enough information around the web about the research and work already that took me weeks to figure out, and months to improve. Most of your work has been on the management/cloud side. Most of ours has been in performance and reliability. So you’ll understand if I’m hesitant to just spell it all out.I’m not saying you’re inaccurate, and that you will for sure have problems, but as a man that I know cares about being honest and accurate you will want to check these things out for yourself and verify. Don’t base your numbers on SLs hardware, their software architecture was honed and profiled/tweaked over a much longer time than we’ve had. Definitely don’t base your numbers on bots.

          • Ilan Tochner says:

            Thank you for your input. Our concurrency claims are currently based mainly on other people’s OpenSim performance research. We’ve had a few occasions were several dozen people visited one Kitely world concurrently, and the sim running that world handled it with ease, but we haven’t had anyone try to get 100 concurrent users in any of our user’s worlds yet.

            We significantly reduced the load caused by asset handling so that using the same OpenSim versions people benchmarked on similar hardware we should get similar concurrency numbers but with avatars and environments that have more assets. I’m confident that we can support 100 concurrent avatars on a single sim in some cases but that there will be other cases were this won’t work. For example, if people create worlds with many physical objects that need to be handled (e.g. many thousands of moving balls, etc.) then that can create significant server load even when there is just one avatar in the sim. It all comes down to how people use their worlds, the more movement they create the more work the server needs to do and the less responsive it will become.

          • It also depends on how many people try to “drop by” at once  🙂

            As always, good luck in all your endeavors. I’m sure there are ways we can more closely collaborate on such things in the future.

          • Ilan Tochner says:

            Thank you Tranquillity, I look forward to that future collaboration.