Kitely attacks slow scripts

There are many reasons why a region might be slow to load, but one of the hardest to address has been that of slow scripts. OpenSim doesn’t make it easy to discover which particular scripts are contributing the most to lag — but that is about to change.

Kitely has fixed a long-standing problem in OpenSim where there was no useful script performance information in either the Statistics Bar or the “Debug” tab in the “Region/Estates” dialog window when users pressed the “Get Top Scripts” button.

Oren Hurvitz

Oren Hurvitz

“Until today, the Script Time shown in the Statistics panel was always 0,” said Oren Hurvitz, Kitely’s  co-founder and VP of R&D, in an announcement earlier this week. “That statistic had never been implemented in OpenSim.”

That data is now available to Kitely users, and region owners should shoot for a value of under 1 ms, he recommended.

Top scripts dialog window. (Image courtesy Kitely.)

Top scripts dialog window. (Image courtesy Kitely.)

“Some worlds might legitimately need more Script Time, if they have many scripts or many animations, but usually if the Script Time is higher than a few ms then you should try to optimize some of the slowest scripts in your world,” he wrote.

The “Get Top Scripts” button now works as well, and shows how much time has been spent on each script in the previous 30 seconds.

The slowest scripts can then be fixed, deleted, or temporarily shut down if, say, there’s a big event going on at the region.

Once this feature has been tested, it will be donated to the OpenSim community, he said.

Ilan Tochner

Ilan Tochner

“We felt this feature was too important to keep private,” Kitely CEO Ilan Tochner told Hypergrid Business. “We’ve therefore decided to help people on all OpenSim grids combat this problem by contributing our latest new feature to the OpenSim project.”

Tochner added that this particular feature is important because of its potential to improve the entire hypergrid.

Related Posts'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China. Follow me on Twitter @MariaKorolov.

25 Responses

  1.' Alex Ferraris says:

    Sorry but we have that in AviWorlds. 🙂 Always saw the scripts there and not zero. Maybe i am not understanding.

    •' XMIR Grid says:

      I was wondering about that too because I also get updated script times in the viewer?

      •' Ilan Tochner says:

        Please read our blog post. There are two places where script-related statistics should be displayed. One is the Statistics windows (CTRL-Alt-1) and the other is the Top Scripts window. OpenSim 0.8.2 doesn’t include an implementation for the former (our patch will add that). While the later (Top Scripts window) statistics are present in OpenSim 0.8.2 but have various implementation problems which greatly reduce their usefulness for finding script load issues. It isn’t enough to show some numbers it’s important that the numbers that are shown will be accurate and actionable.

        •' XMIR Grid says:

          I have no doubt the figures in Top Scripts are inaccurate, but they at least gave you an indication. I was amiss because the article said they were always displayed as zero.

          Any work that updates the accuracy and usefulness of the statistics is highly welcome, so we look forward to the submission being implemented in standard OpenSim. Great work!

          •' Ilan Tochner says:

            The article quotes “Until today, the Script Time shown in the Statistics panel was always 0” – that is the Statistics window you open with CTRL-Alt-1 which indeed doesn’t include script runtime in OpenSim 0.8.2 (try it now on your own grid).

            One of the problems with the Top Scripts implementation in OpenSim 0.8.2 is that it didn’t include indicative data about all the scripts that require the most amount of time to run. For example, a script that takes a long time to run (and creates lag) but is called only once per 15 seconds may not get included in the top scripts list under certain conditions. Alternatively, a script that takes very little time to run but is called thousands of times per second (and it too creates lag) may not be included at all because each call by itself is insignificant compared to the time it takes to handle the other scripts being called in the sim. Our patch addresses these scenarios and more.

    •' Bryan French says:

      We are talking about 0.8.2 aren’t we? I have found the Top Scripts
      window to be very accurate, but we aren’t running 0.8.2 either. In my
      Statistics window it does say Scripts Run 0.000%, but it does say on the statistics there are
      68 scripts running and all 68 show up on the Top Scripts even the ones running 0. When it comes to scripts on our grid we will only allow scripts we have tested and that run under 1.0ms. That is why you will not find a blink script on our grid. They run continuously and over 1.0ms

      •' Ilan Tochner says:

        If you look at the OpenSim code that implements this panel you’ll see that what it currently calculates for Top Scripts is far from accurate. The fact that many people currently trust what they see is one of the main reasons we decided to contribute this patch.

  2.' TribeGadgets says:

    Nice add but is your welcome centre ‘which is a highly optimized world’ really running this Total Frame Time? 88ms+ (quote and snip from your blog)
    I would worry if my ancient box running a small 0.8x simona was that bad.

    •' Ilan Tochner says:

      OpenSim 0.8.2 includes patches from the MOSES project which significantly improve the statistics you see in your viewer. The frame time you see reflects the actual Sim FPS that OpenSim runs at. The numbers you see if you are running OpenSim or earlier are multiplied by a constant that the developer that first implemented stats added in order for them to look like the numbers people were used to in Second Life.

      In other words, the stats in our screenshot represent the maximum Sim FPS you will see once you switch to OpenSim 0.8.2 (unless OpenSim core decides to increase it). In any case, the base Sim FPS hasn’t changed it is just that now the Statistics panel shows the accurate numbers your sim is running at.

      •' TribeGadgets says:

        So wheres the ‘missing time’ here then 🙂

        Curious as spare time is 18.2 🙂
        Do like expanded info so yep.

        •' Ilan Tochner says:

          Sim FPS in OpenSim is currently set at 11. That’s the ractual rate all OpenSim versions are running at. As indicated in the screenshots that leaves a lot of spare time on this particular sim.

      •' XMIR Grid says:

        > OpenSim 0.8.2 includes patches from the MOSES project which significantly improve the statistics

        Can you point me to where these are because to my knowledge they still only exist as patches attached to Mantis records and have not been committed to the code base. Of course Kitley may have deployed these, but it is not generally available.

        •' Ilan Tochner says:

          You can find the ones we integrated in OpenSim’s github repository.

          •' XMIR Grid says:

            None of the MOSES patches have been integrated in OpenSim core yet. It is still being discussed. They exist in MOSES’s Git repository.
            I certainly hope they will be integrated in core. 🙂

          •' Ilan Tochner says:

            They sent 3 patches to date. The earlier ones are in OpenSim 0.8.2 IIRC.

          •' XMIR Grid says:

            Correct, they sent 3 patches and the first one being committed slipped under my radar. It was committed at the end of April under “Added in all metrics for Phase 1 except for Time Dilation.”
            I think I tired paying attention to it after a good deal of fuzz on the dev mailing list.

            Not sure where phase 2 is, and phase 3, which is the network stats, is still under review after they submitted the final patch on August 5.

            Thanks for helping clarifying that! 😉

      •' XMIR Grid says:

        It would be interesting to know from you who have tested it where you think the frame time should be at?

        After observing simulators they run at around 95 ms ± 2ms which ends up at pretty much the same FPS as Physics FPS in the stats display. (1000 ms / 95 ms = 10.52)

        •' Ilan Tochner says:

          It’s a function of the FPS you wish tho have. At 11 FPS you get around 90.9ms per frame. If the FPS is a bit higher (around 11.2-11.3 FPS) then you get the numbers you see in the screenshot, i.e about 88.9ms per frame. If you read historic conversations in the OpenSim dev mailing list you’ll see that the OpenSim target FPS was arbitrarily decided years ago. I’m quite confident it could be increased, but while doing so would increase system load, I’m not sure it would contribute much to actual user experience (people seem to be happy with OpenSim’s performance as it is).

          •' XMIR Grid says:

            I agree with you that performance in general is where people are happy about it (at least when the dust have settled and they are rezzed in to a scene.)

            I was just a bit puzzled that the new Frame Time FPS produced a number almost identical to the Physics FPS. I was under the impression that (as you say) the desired Physics FPS is set at a 11 target, but that the scene FPS was independent of the Physics FPS.

          •' XMIR Grid says:

            Thanks for the refresh. I remember having read it before, I just had not entirely realized that Physics PFS = Sim FPS with a target heartbeat at 11 for OpenSim.

            From observing the sims a bit more I see they fluctuate between 89 and 98 ms where at the high end they probably stall a bit on mesh loading and decoding. (meshes are always physics unless set to phantom).

            I run them on mono and one of the things about mono 4.x is that it is pretty aggressive on purging memory so perhaps I should look into that. Simulators that typically ran around 2GB with mono 3.10 now run with process memory of less than 512 Mb.

          •' Ilan Tochner says:

            Yes, having aggressive garbage collection can cause frequent stalling. 🙂

          •' XMIR Grid says:

            I guess the reason why LL has let the physics component for meshes weigh as much as it does in the LI calculation is because they have exactly the same relation between physics simulation and Sim FPS. They just don’t break it out in their stats much.

            Actually on the marketplace there should be tick marks for if an item is 100% mesh, partial mesh and if it was uploaded with a separate physics model or not. The difference on the sim can be quite substantial just for a single mesh if it has an optimized physics model over the “default” generated out of the mesh geometry at upload.

          •' Minethereé says:

            now see what you have done is show me a good place to watch drama unfold!! thank you!!! oh lordy, lordy I desperately needed something new to read……………………….


  3.' XMIR Grid says:

    Built, tested and deployed Oren’s code drop today, and it all looks hunky-dory. Thanks a lot Kitely for bringing this!