E-Mail 'Kitely battles slow-loading meshes' To A Friend

By:
On:
Tagged:

Email a copy of 'Kitely battles slow-loading meshes' to a friend

* Required Field






Separate multiple entries with a comma. Maximum 5 entries.



Separate multiple entries with a comma. Maximum 5 entries.


E-Mail Image Verification

Loading ... Loading ...

47 Comments

  1. fonsecaloffpt@yahoo.com'

    As a world builder this is very useful but there should be a way to test if some object is that laggy or not before using it – One could spend hours and hours and hours using meshes that must be non phantom (mountains or buildings to be walked upon) and only afterwards find out they are useless unless phantom – And worst, one can buy something without knowing is bad – I hope someone in OS finds a way to test stuff first so builders and creators do not get into stress and frustration, that is not good for OS

    1. ilan@kitely.com'

      If you’re creating meshes using Blender then simply use its built-in tools to create a version of the mesh with a reduced triangle count to serve as that mesh’s physics mesh. Any other half-decent mesh creation application will enable you to easily do this as well.

      The overwhelming majority of meshes we’ve encountered inworld take only a few milliseconds for OpenSim to process so the 1.5 seconds threshold we’ve set for our system is hundreds of times higher than what most meshes actually require. There really is no good reason for using a very high triangle count mesh for physics, doing so is like placing a 5MB uncompressed BMP image on the front page of your website instead of simply saving it as a 100KB compressed JPEG file first. In other words, if you make it a habit of following the simple aforementioned procedure then you won’t create overly-complex physics meshes that our system will later need to turn into Phantom.

      If you bought a mesh in Kitely Market and it turns out to have an overly-complex physics mesh then contact the merchant who sold it and request that they optimize their product’s physics mesh and Redeliver the updated product to you using Kitely Market’s built-in redelivery feature. If they refuse to do so then write a bad review for their product listing and use the Report link on that listing to let Kitely know about the merchant’s refusal to fix the problem with their product. As Ozwell stated in the article, “It is absolutely the merchants responsibility to make sure what they sell is optimized to the best of their ability”.

      1. brayla@braylasana.com'

        In addition, many small objects (candles, pillows, etc) can use the simplest cube for physics. You can use the same cubes over and over.

  2. mike.dickson@rivendellnh.com'

    The testing is, I think, called LI. Integrated into the upload process you get feedback before the upload of the LI of your mesh so you can choose to optimize it further if needed. There’s code in Halcyon to implement both a reasonable LI calculation and to provide the feedback during the upload process if anyone would like to look at how it’s done there.

    1. fonsecaloffpt@yahoo.com'

      Does that LI calculation tell us what it translates into miliseconds server upload ? Cause just knowing the LI would not be enough to know if I was using a safe freebie or buliding a safe creation or buying a safe product – When I said testing I meant exactly to test what is at stake – Miliseconds region startup

      1. mike.dickson@rivendellnh.com'

        Region startup time is going to be based on lots of factors. How the region is sized and what its running on, etc. LI (Land Impact) should be a measure of complexity not an absolute value like milliseconds startup. The problem is OpenSIm has historically valued Mesh at LI=1 regardless of how complex a mesh actually is. So there is little incentive for anyone to optimize them at all. The physics mesh is just one part of course. There is storage for large mesh data and the download cost to get it to all the clients in the region.

        1. ilan@kitely.com'

          Kitely’s new system is designed to address complex physics meshes and the effect they have on world startup times, not any other contributor to bad world performance. LI for a tiny object is smaller than the LI for a big object even if they both use the same mesh. That’s one of the reasons why LI is less useful for measuring what the effects of a physics mesh is on startup time than the actual effect a physics mesh has on startup time. There is no point in using a proxy to measure a parameter when the parameter can be measured directly.

  3. b54563@mvrht.com'

    Not really too pleased about this decision myself. I don’t like that it was forced it should be a setting that can be toggled per region by the owner if they care about the loading time. Its a bit concerning because so much content has been purchased and its not as easy as contacting the people to replace the items many just don’t respond or want to take the time to fix the items. Reason why I have the max package with kitely is to not have to worry about this but may have to start looking at alternative hosting spaces because one of the reason coming to opensim is to not to have to worry about limits. May have to try to give running OpenSim a spin for myself, just don’t want to have to worry about this stuff. I have only been buying exportable stuff anyways so I should hope that by saving the oars I can just take elsewhere. Maybe I should have went to the meetings, I didn’t realize this was even coming.

    1. ilan@kitely.com'

      Hi Thelma,

      First, you’re welcome to join the weekly Kitely community meetings, every feature we roll out is discussed in them beforehand. See: https://www.kitely.com/virtual-world/Ilan-Tochner/Kitely-Welcome-Center

      Have you tried entering your Kitely world and seeing how many of the objects you acquired were actually affected by this? I doubt it was more than a handful (if any) as the worst case we’ve seen so far only had 21 objects (out of more than 40,000 rezzed in that world) that were affected by this, and almost all of them were freebies that were brought over from other grids (i.e. not items bought in our marketplace).

      Kitely’s new system saves you from having to worry about complex mesh objects slowing down your world, without it you can have a degraded level of performance and have a hard time tracking down what is causing it. If you didn’t experience slow world startups then it’s unlikely that this new system needed to do anything in your world.

      1. b54563@mvrht.com'

        Only a few items for me have been an issue, I just wish that this was optional at the region setting level. I do understand the stance and why its being done, but I don’t like being forced in to change. Nobody does.

        1. lmpierce@alcancemas.com'

          I agree that a switch for this feature would be great. For a private closed region, the creator may enjoy using their virtual world for viewing mesh objects of varying complexity, and the slow startup may not be an issue to that individual.

          1. ilan@kitely.com'

            This doesn’t prevent uploading or viewing of complex meshes it only prevents using overly complex physics meshes for them.

            I think the people who feel this is being forced on them don’t quite understand how badly optimized a physics mesh needs to be for this feature to affect it. There really is no good reason for having physics meshes be that complex, even for objects with holes, tunnels, arches, etc.

          2. lmpierce@alcancemas.com'

            So this is the point on which my understanding vacillated. First, from the comments only, I thought it was ‘physics’ meshes, and as you say, they can be relatively simple compared to the visible object. But then, on reading the article, it was said that a ‘complex mesh object’ could slow things down on startup, which suggested that visible mesh complexity was also an issue. I can see why this could be confusing.

            As for the diesel truck analogy, I do take issue with the extreme comparison. On the Earth with nearly uncountable and often times unaccounted for egregious forms of waste, there is no need to guilt the tiny band of OpenSim users into being more efficient with their physics meshes. The tiny difference is moot. If I forget to turn off my patio light tonight, it will probably waste more electricity than my poorly designed mesh physics objects might waste in a year. In fact, I would posit that being ‘wasteful’ in a virtual world is probably one of the most benign forms of inefficiency I can think of. But I would emphasize that point by a factor of 1000 or more considering Kitely’s on-demand model. We already use such a tiny fraction of electricity compared to Second Life or other ‘always on’ services, I feel it is inappropriate to play the wasting electricity card.

            Rather, I would think that your point about load times was enough to drive concerns and motivate greater efficiency. I can’t imagine any owner that wants to wait 5 or 10 minutes to enter their world. But most of all, your clarification about the physics mesh versus the object mesh reiterates the core issue, which I think is helpful.

          3. ilan@kitely.com'

            I now understand your previous resistance to this feature. To make things absolutely clear for others as well, I’ll restate that Kitely’s automatic system doesn’t change anything visible in your world. Our new system just helps prevent parts of the invisible things, specifically the physics meshes that are used by objects, from slowing down people’s worlds.

            As for the pollution analogy, the effect on the real world environment may be tiny but the effect on other people’s worlds from someone’s world maxing out its CPU usage for an extended amount of time can be very real in some extreme cases. This negative effect is limited to no more than 3 other worlds in Kitely, depending on their respective world types, but doing so knowingly is still very close to knowingly blowing smoke in other people’s faces. As a service provider hosting more than 11000 user worlds we need to ensure that this doesn’t happen and not leave it as an option for our customers to decide whether or not they care enough about this to do something about it.

          4. b54563@mvrht.com'

            This may be a stupid question but could it be that maybe the servers are not powerful enough? If this is so maybe we can have our regions upgraded to something more powerful?

          5. ilan@kitely.com'

            I apologize for the delayed response. There are various things that OpenSim doesn’t handle concurrently (that is it can’t split the work of handling them by dividing it between multiple CPU cores and must have a single CPU core handle the entire process, something that is referred to as a single-threaded calculation). One example of single-threaded calculations in OpenSim is the parsing of physics meshes where each mesh is handled entirely by a single CPU core. In other words, the speed that OpenSim can parse a physics mesh is limited by the speed of a single CPU core.

            Servers nowadays scale mostly by way of increasing the number of CPU cores they have and not by making each CPU core work significantly faster than the CPU cores of slower servers. This means that upgrading from one modern server class (such as those that Kitely uses) to a more powerful server class increases the number of CPU cores available to your applications but doesn’t have a significant effect on calculations that are single threaded.

            In other words, upgrading our already powerful servers to even more powerful servers (which derive most of that extra power by having more CPU cores not faster ones) will not really resolve the parsing overly-complex physics meshes problem that our new system is designed to address.

  4. ozwellwayfarer@gmail.com'

    Hey Maria

    Could you please not use an image of my meshes as the thumbnail image for this article?

    I dont want people thinking that Worlds End products will be affected by this change, because categorically, they will not.

    1. maria@tromblyinternational.com'

      Sure. I’d put them in to illustrate your quote, but I can replace them with some other mesh picture.

      1. ozwellwayfarer@gmail.com'

        Thank you! Sorry to ask, but its not so cool having this headline and my meshes displayed on things like FB and G+.

        1. b54563@mvrht.com'

          Free publicity?

          1. ozwellwayfarer@gmail.com'

            Thanks Thelma 🙂 I guess they do say there is no such thing as bad publicity. lol

  5. tribe_gadgets@gmx.com'

    Threshold seems reasonable enough I guess. But I always – always – test mesh stuff out first on SL before it ever gets placed anywhere else precisely because the uploader gives me full feedback on impact for the physics engine there as well as data on the LOD optimise/return mix. (Saves me a little bit of cash too in that case which is not an issue elsewhere). Question is why not implement the same feedback at the upload stage in Kitely?

    1. ilan@kitely.com'

      Unlike in SL where every new mesh to arrive into the grid is via the viewer’s upload tools, in OpenSim content can arrive via OAR imports and the Hypergrid as well. This means that it wouldn’t have been enough to notify people when they upload meshes via the viewer, we would have had to create tools for all the other options as well. Even if we’d done so, we would have still needed to deal with all the overly complex physics meshes that have been brought into Kitely over the years. The simplest solution was therefore to address the problem directly and deal with the meshes that cause problems automatically instead of creating a bunch of new manual tools and forcing people to learn how to use them.

      1. tribe_gadgets@gmx.com'

        No skin off my back, was merely clarifying so I can answer those who I make stuff for, I can understand that things come in from ‘elsewhere’. I make a habit of optimising via the constraints imposed on me as I have learned them. Yes I too have seen the hi-poly monsters on various grids and.. *shudders*
        That being said, it is very easy to hit ‘use above’ all through the mesh upload process with no apparent impact. Which, as i understand uploaders although I may be wrong, results in a de facto phys model assigned. And I do upload to where items are to be used so I have a reference.
        I have a link to this to help explain stuff to people who asked me so all good.

  6. dahliatrimble@gmail.com'

    While I agree that overly complex physics colliders in meshes are in general a bad thing and should be avoided, I think an alternative/adjunct to the phantom threshold could be to prioritize mesh loading so that the first to load are those most likely to be involved in collisions (as in closest to avatars or physical objects). This is something I was once considering implementing, but then again, I’ve never seen any mesh asset take more than a few seconds to load on my development grid, even when the colliders are at the limits of complexity.

    There’s another issue at play here. Very complex mesh assets can be very large in memory size and OpenSimulator typically sends the entire asset to viewers. I remember there was some discussion among Linden engineers that viewers could ask for only the portions of the asset that they needed but I don’t know if that feature ever made it into the protocol, or if it had, if it made it into OpenSimulator since I left. I also don’t know anything about how Kitely stores and distributes their assets or how the region server is involved there. Generic OpenSimulator would default to caching mesh assets for each region which would help quite a bit to reduce startup time and time it takes to deliver them to viewers.

    1. ilan@kitely.com'

      Hi dahlia,

      Kitely caches physics meshes to avoid recalculating them every time the world starts. The problem is that it can take OpenSim a long time to even just load a cached physics mesh if that physics mesh is very complex. In most cases this loading process takes mere milliseconds but in some cases, such as the ones our new system is designed to address, it can take many orders of magnitude more time to do so (up to dozens of minutes for loading a single absurdly complex mesh that fit in memory but just barely).

      Pathological cases aside, most Kitely worlds (even ones with tens of thousands of objects and thousands of scripts) start within less than a minute (i.e. significantly faster than standard OpenSim). The problem is that it only takes a single badly optimized mesh to significantly slow down the process. Our new system is designed to eliminate those outliers so that all Kitely worlds will start up quickly.

      1. geir.noklebye@dayturn.com'

        it would be interesting to see such an absurdly complex mesh.
        How did it even enter the grid as the viewers (are supposed to) limit the size, and for viewers like Kokua and others that have newer Linden code for the mesh uploader any complex mesh is split not a series of smaller sections that should load efficiently?

        1. ilan@kitely.com'

          We’ve seen complex meshes inworld with hundreds of thousands of triangles where the physics mesh was the same as the mesh itself. Most such criminally complex physics meshes arrive to our grid via OAR imports and the Hypergrid so I can’t really say how they are imported inworld.

          1. geir.noklebye@dayturn.com'

            Sounds like high resolution models that is not adapted for use in a gaming environment where the model itself is uploaded as the physics model. These are often ripped or copyboted commercial models. There are multiple alternatives for how they go into the grid:

            1. The user deliberately use the model itself as the physics model.
            A viewer with updated code will not do this but will generate a convex hull, meaning the model cannot be walked inside, or on in many cases. This usually happens if the uploader did not create the mesh themselves as they very fast will learn how to make a simplified physics model for their models.

            2. The uploader is using an old viewer that for some reason pics the model mesh as the physics model if the user does not specify anything else in the upload pane.

            3. The viewer is a self compiled viewer with modifications to “automate” uploads or circumvent limitations set in the original LL code.

            4. The viewer is a copybot viewer that let you upload just about anything as long as the server don’t stop it.

          2. ilan@kitely.com'

            Thank you for the overview. Whichever case may be, we still need to deal with such objects when they are imported into our grid via the Hypergrid or OAR files. That means that we can’t trust that the problem will be prevented by other means outside our control.

  7. skylifegrid@gmail.com'

    A lot of people upload poorly created or obscene amount of polygons from sketchup or the likes. This is always been issues with free uploads people will really go to town and someone without any experience in low poly design can bog down the system real quick

    1. chrisx84@live.ca'

      allow me to explain this “low poly” you speak of into terms that people can understand 🙂
      These “poly’s” are triangles, each square generates 2 of these so 1 square equals to 2 poly’s.
      Peoples video cards (and the intel chipset) see’s these poly’s as data because that is what they are when sent from the central processing unit (CPU) to the Graphical Processing Unit (GPU), its the GPU that converts these poly codes into something 3D.
      So if you got a cube with just 2 per side, everyone’s video card will be able to process it in just afew miliseconds.
      If a cube has like 20,000 polys then it will take some time for the gpu to convert each poly code into a 3D object.
      This is why if you want the best visual experience in SL/Opensim its best to go spend the money on a super fast video card, i say at least a 2GHz and at least 3 GB of ram since the GPU needs that ram to store the poly data as well as texture data.
      But remember when meshing always keep the polycount as low as possible, only create what you need for your mesh.
      Any extra should be deleted. To do this in blender is super easy, find and select a useless edge and press G twice then move that edge to a nearby edge, repeat the process till theres no more useless edges since that can generate poly’s, then select Remove Doubles in the left panel (T menu) and BOOM, low polycount.
      I really hope that helps alot of people understand polycount.

    1. ilan@kitely.com'

      That’s exactly the problem our new system is designed to prevent. People have seemingly innocent looking objects that are slowing down their world without them even knowing about it.

  9. Now I know why when I moved a var region from my beta grid to kitely that lag was awful. The worst I’ve seen anywhere. The meshes I used where from the Kitely market and well known mesh creators (so not brought in from the hypergrid).

    My grid didn’t struggle or show any signs of lag at all. So I’m not sure if I agree with the assumption that this is something Opensim is responsible for.

    I have move that region back to my grid and it’s back to normal. I also follow the design recommendations.

    1. b54563@mvrht.com'

      I’m beginning to wonder if maybe the servers are just not powerful enough for the mesh because my oar works everywhere else with no issues.

    2. ilan@kitely.com'

      Prove it by posting a link to one of your Kitely Market orders that caused you issues in Kitely or anywhere else. Let’s see if your story holds up or if you’re just a troll seeking to spread lies.

      1. Well Ilan thank you for a wonderful example of how not to respond to a post. I am actually suggesting that it may be something to do with Kitely as a platform. I have a bare metal server with 32GB of RAM, an i7 processor and 500GB SSD Drives in it.

        My var with the well built and optimized mesh (three highly respected mesh builders whose content I used specifically because of that reputation) from Kitely Market shows no signs of lag AT ALL. There is nothing else on that var region from the hypergrid so the likelihood of these meshes creating the problem is highly suspect as is the assumption that it is Opensim. ITS YOUR SYSTEM DOING IT. I loaded the same region into Kitely and I could barely move. That is what I was suggesting in my post you so rudely responded too by calling me a troll. This is why you can stick Kitely.

        Don’t bother responding Ilan as I dont talk with rude people particularly when I am paying you money.

        1. ilan@kitely.com'

          I understand what you’re claiming I’m just stating that your story sounds dubious to me. If you are a real Kitely Market customer then you can post a link to a Kitely Market order page to prove that you bought an item in our marketplace. I can then check what Kitely worlds, if any, you’ve had and see if they indeed had a high server load or not. If they did then we can check which items caused the load you experienced.

          Regarding your personal hardware, I venture that you’re not getting it from a hosting company, as that would cost you a hell of a lot more than what Kitely charges. You shouldn’t need to have such a powerful computer just to run OpenSim. There are very complex builds that run just fine on hardware that is a lot less powerful.

          For comparison, Kitely places up to 4 worlds per m3.large EC2 instance in the Amazon West datacenter (depending on their world types and current concurrences). Each Kitely world gets between 1 and 4 GB of dedicated memory. We have worlds with 100,000 prims and thousands of scripts running on Kitely just fine even on a 2GB world configuration, for example: https://www.kitely.com/virtual-world/Lawrence-Pierce/Serenity-Island-Prim-Capability-Test-2 .

          You can compare our hardware and software offering with that of alternative hosting companies and grids to determine just how much more bang for the back you get with Kitely. If your build was laggy in Kitely then I suspect that the issue wasn’t due to server load. If you ever test that again I recommend you use our built-in Performance Monitor to check server load and see whether the issue has anything to do with server lag: https://www.kitely.com/virtual-world-news/2015/08/16/monitor-world-performance/

          1. I am a real customer Ilan, Leighton Marjoram. Its not the content as i stated previously. It’s a quirk to your system and its limits and its for you to find a solution to, own it as such not call your customers out in public and calling them trolls because of the limits in your own business model.

          2. ilan@kitely.com'

            I just sent the real Leighton Marjoram an email to the address we have on record. If he acknowledges that you are really him then I’ll make a public apology to you and work to investigate the build you said caused issues when uploaded to our grid.

          3. My identity has been confirmed and do I deserve my apology Ilan …..

          4. ilan@kitely.com'

            Your apology was given once I finished writing the following:

            First, I apologize for claiming you are a troll, we’ve confirmed that you’re not.

            I replied as I have because I know what causes server lag
            and it has nothing to do with our hardware or software backend. I’m
            frankly quite fed up with people making anonymous statements without
            providing any actual proof. You can read our forums to see that I’m very
            patient and helpful to people who report concrete issues, even when it’s very clear that they don’t
            result from any shortcomings in our system.

            I’d be more than happy to
            provide you with enough Kitely Credits to upload your build into a Kitely Standard World
            and check how well our system handles it, and what causes lag (if anything). I’m sure that if there is server lag then you’d benefit from knowing it’s source and eliminating it even if you intend to delete that build from Kitely once you’ve learned that information. Your powerful hardware setup may be hiding a performance issue from you, one that will become apparent once your grid gets a high enough concurrency. For your own grid’s performance I recommend you take advantage of my offer and our performance monitoring and optimization tools to improve your build.

          5. Thank you Ilan for the apology. Looking back I should of re read the tone of my original post. I was surprised that the performance was the way it was. I have a mentor that I can ask to look at my server as I do wonder if the server as you say is masking something.

            Note to self re read posts for tone before hitting send.

          6. ilan@kitely.com'

            For my part, I’ll make sure to reread my posts as well. I shouldn’t be commenting on blog posts on a Friday evening while I’m entertaining company inRL.

            I really do recommend you take me up on my offer. Our tools will help you track things that can be hard to find when running a standard OpenSim setup. For starters, Top Scripts and Top Colliders don’t perform as well in plain OpenSim as they do in Kitely: https://www.kitely.com/virtual-world-news/2014/01/22/scripting-in-advanced-megaregions-and-troubleshooting-physics/

            Second our performance monitoring tools can help you track load visually over time: https://kitely.atlassian.net/wiki/display/doc/Optimize+World+Performance

            Third, our new system will help eliminate overly complex collision meshes from your build, those can take a tool on inworld performance when you interact with them and that can be hard to trace manually.

            Finally, our logs contain other information that isn’t included in standard OpenSim releases.

        2. ozwellwayfarer@gmail.com'

          Wow Ilan, really? No need.

          Leighton – It was mentioned in the comments of the Kitely Blog that it is possible for a world to share a server with other worlds.

          I imagine if you are using your own dedicated server, you will naturally get better performance. I looked up the rough price of renting 1 server with the specifications you mentioned and its roughly in the 50+ dollar range.

          The price of a 16 region world in Kitely is 39.95

          So I think its just a simple case of economics at play here. The server you mention is expensive to start with and Kitely offers more than just a server to host opensim. There are all the improvements that have been made, the kitely market, grid services, community management and customer service, the website etc. That all costs money and somewhere a profit needs to be made too.

          If your happy to just run Opensim on your own server and dont feel the need for the added value Kitely offers, then maybe that is the best option for you.

          1. That’s a little more reasonable response. I think maybe the comparison is a little off as comparing always on dedicated server to an on demand VPS deployment there will be differences. As a matter of fact the server is about £60 a month so its really not that expensive.

            You are one of the content creators whose content I always use as I know that you are a skilled content creator and knowledgeable about preparing and optimizing mesh for virtual worlds. The others I will not mention as I am not happy that Ilan is calling their work out as the problem when I clearly stated otherwise.

            I am sticking with my own hosting as the resource restrictions which make perfect sense as a business impose limits on what can be done. Thank you for you more considered response.

Comments are closed.