Upcoming changes to monthly stats report

I’ve been doing the stats report every month — without a break — since mid-2009. With nearly 300 active grids, many of which don’t have standard stats pages, the amount of work going into this has become unmanageable. It eats up parts of several days of my time, contacting everyone, collecting the stats, analyzing them and writing them up, then fixing any errors and updating data from grids that only respond to my emails after the report has already been published.

But I’ve also received emails from folks who find it valuable. Plus, I also use the opportunity to update the active grids list and the list of most popular hypergrid destinations. These are useful not just for regular users, but also for viewer developers and hyperport operators.

Another issue is that of helping grids get found by search engines. Each stats report has links to the most popular grids, to the largest grids, and to short news announcements.

We’re currently getting more than a quarter million page views a month, which is good for increasing the value of those links. Plus, Hypergrid Business is indexed by Google News — we had to go through a whole approval process. As a result, we’re considered an official news site, and more authoritative than the average site of the same size.

So here is what I’m going to do.

First, I’m going to stop collecting data from grids where I can’t automate the process.

I’ll run a database script that will check whether grids are active or not by going to their grid info pages. For example, here is the one for 3rd Rock Grid: http://grid.3rdrockgrid.com:8002/get_grid_info. I’ll use that to update the list of active grids.

I’ll also run a database script to collect data from all the grids where I can scrape the stats. Some grids do weird stuff with tables, or use periods as a thousands separator, or do other stuff that makes data collection very difficult.

The database can do that in the background, while I do something else, and I’ll just collect the reports.

Second, I’ll send the data — and forward all news announcements — to David Kariuki. He’s been writing a lot about OpenSim lately, and is starting to learn his way around. You can also email him directly at [email protected].

Fixing stats pages

Here’s an example of a stats page that works very well: http://members.greatcanadiangrid.ca/GcgStats.php

It shows total registered users, active 30-day users, and the total land area — and no extra stuff to slow the database down. It’s just what I need.

Hypergrid visitors should be included in the total active user count, and the total land area can be in square meters or kilometers, or in the form of standard region equivalents.

Wifi pages also work well. Like this one: http://login.ansky.ca:8002/wifi. The one problem is that they don’t show the total land area, which means that grids with varregions will be undercounted.

Here are the five biggest grids where I have problems automating the stats:

  • OSgrid: Doesn’t show total land area
  • Metropolis: Periods in the numbers where a comma should be
  • Kitely: Doesn’t show active users
  • YrGrid: Has JSON export that I need to load in the browser
  • Lost Paradise: Weird formatting
  • 3rd Life Grid: Strange HTML that locks up my database

Currently, about a third of all grids have some kind of issue with their stats. Incomplete data, inaccurate data, and assorted weirdnesses of various kinds.

If you have a question about whether my database can read your stats page, or to send me a link to a new stats page, email me at [email protected]

Changing site focus

I’ve made some changes to the site layout and editorial focus over the past year, and will be doing more of that.

I originally got into covering OpenSim because I saw it as the most direct and likely path to an open-source, peer-to-peer, hyper-linked virtual reality metaverse. I’m not so much of a believer in OpenSim as a believer in its potential. I don’t want to live in a metaverse owned by Microsoft or Facebook. I want to be able to create my own virtual world.

When virtual reality finally came along, I was super excited. I few to San Francisco for the conferences, met the guys — and gals — creating the new technology, and tried out every headset I could get my hands on. And I was anxious to start using using this hardware to explore OpenSim. I even got it to run on my smartphone last summer.

There were some complaints, but overall the majority of OpenSim users were with me. (And I ran a survey just to check.)

But then something weird happened last few months. The headset reviews caught on. Traffic exploded. Today, most of my readers come for the virtual reality hardware reviews. I had more than 137,000 unique readers over the past 30 days, which is about 100,000 more people than there are active users of OpenSim.

I’m excited about that, and want to do more of it.

And if anyone is interested in joining me, you know where to find me.

Related Posts

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China. Follow me on Twitter @MariaKorolov.

  • Da Hayward

    Good stuff Maria. Personally I think you have done a more than great job and would like to think that most appreciate the efforts you have put into this. Even the Know at all’s who always get anti re figures because they are too lazy or self important to contact you.

  • Tony Anytime

    Yep, right on. “A grid e byedMicrosoft or Facebook”, death by be. Being nice….. But hang in there, the metaverse is going to get interesting soon and you are sitting on your own rocketship. On stats, I would go further. As you said the grid’s benefit from the exposure, put it on them. Make them generate a stat file of your format. Kiss, but you design it,. Conversely for the small grid’s that don’t have the tech, or it doesn’t work for them at the moment, have a webpage that they can fill out. But if they don’t do either they get a no stats this month entry.

    An interesting thing would be to put the stats into a module into opensim. May be next version… Hint hint

  • Talla Adam

    Thank you Maria for keeping the stats list going for all these years. I am certain most Opensim users appreciate it. I know myself just how time consuming and difficult it is to collect the data since, as you know, I built a grid search engine a few years back and I came up against all the issues you did. I also found I had to do a lot of work manually to try and get the numbers right and I honestly didn’t have time for that, running my own SL regions, Farworldz grid and the G+ Opensim Virtual community. So, I have to thank you for all your work and also Magnuz Binder for his grid stats list too.

    And Merry Christmas to you Maria and the whole Opensim community.

  • Magnuz Binder

    Just a small correction: “Metropolis: Periods in the numbers where a comma should be”.

    Metropolis, being a mainly German grid, actually follows the original and general rule for separators in Arabic numbers, with a decimal comma and a point as thousands separator.

    It’s really just the UK, some of its former colonies, and a few countries heavily influenced by one of its former North-American colonies, at the time they adapted the Arabic numerical system, that use the decimal point and a comma as thousand separator.

    If either the English hadn’t stubbornly stuck to that “everything the French do is wrong” after England lost the Hundred Years’ War (1337-1453) to France (be it metric system, decimal system or thousands separator), or the American rebels had seen the light and adapted proper practice when the French won the American Revolutionary War against the British for them, the problem with _some_ North-Americans (actually, French Canada, Cuba, Haiti, Costa Rica and Greenland do it right) using the wrong separators wouldn’t exist. *winks at the American chauvinism about what “should” be*

    • as an American I bristle at your words!! (goes to look up bristle)…

    • I understand its a regional thing, but there are a lot of German grids out there — I’ve had to add the German words for “region,” “users,” and “actives” to my database (it’s “Regionen,” “Nutzer insgesamt,” and “Aktive Nutzer (letzte 30 Tage),” in case you’re interested.)

      Metropolis is the only one with the periods and commas switched.

      Some use no separators at all, some use the U.S. style, and many never get above 999 of anything so the issue doesn’t come up!

      What I can do is add a section of code to my database that checks if the grid is Metropolis, then strips out the excess period. That will slow down data collection — I’ll have to check every grid to see if it’s Metropolis or not. And it will mean that I’ll have to edit code over the holidays. But the again, I’ll have to edit code anyway, so I guess it’s no biggie. It would be nice if they fixed it, though, because it’s not just me — other people might want to get the stats to, either as a double-check on mine, or to put together their own lists (say, for hyperports) or for their own evil purposes (bwa ha ha ha).

      • “evil purposes” egadssss! but how about everyone else change theirs to how Metro reports and all will be just fine-))))))

        • Magnuz Binder

          I’d be all for the Lost Paradise HTML formatting (agreeing with Maria it is weird and requires a pretty complex sed to straighten out) with the Japan Open Grid stats labels (how about ハイパーグリッド アバター) and Metropolis number format (with proper decimal commas).

          • You’re a better programmer than me, Magnuz, if you can pull out the Lost Paradise data!

            (Of course, most people are better programmers than me!)

            For the Japanese grids, I actually don’t have a problem. In my database, I save the relevant keywords for each grid individually. So I go to the stats page and run it through Google Translate to see where the keywords are located. Then I copy-and-paste the original Japanese text into my database. I do the same for all the languages where I can’t just guess.

          • I guess I can do a general-purpose check… if the stat is supposed to be an integer, and instead there are three digits past the decimal point, multiply by 1,000….

          • Magnuz Binder

            Since one of the main aims with my stats is collecting concurrency figures, which requires a pretty high sampling rate, manual labor was never an option, but I needed to automate the sampling from the beginning, including finding ways to scrape even “strange” stats pages. So far, Lost Paradise and ATEK grid were the most challenging ones, but quite doable with my general approach.

            I don’t know exactly how you scrape the stats from the pages, but if it’s any help, I can describe my general approach, which at present leaves me with 12 out of 343 grids (3.5%) I need to use two lines each of special code to handle. Some of the cases are identical, so it’s just 8 special cases covering those 12 grids.

            I use shell script to strip the pages from linebreaks, HTML tags, extra whitespaces and thousand delimiters, then just grep for stats labels with trailing figures and strip the labels. A few pages I need to reformat before the stripping, a few afterwards, to keep the grepping standard and simple. I’ve tested the system with sampling up to 1800 grids every half hour, which is the sampling frequency I use, so the present 343 “active” grids don’t make it sweat much. Actually, it’s the TCP/IP stack rather than CPU or memory which sets the limit, so I probably could tweak it to handle even more.

            As an example, which might be of help then, the (Ubuntu shell) command I use to reformat Lost Paradise stats page into the standard format is (I wonder how Disqus will manage this one):

            sed ‘s/Total Regions[ :]*Total Members[ :]*World Estates[ :]*Member Groups[ :]*Avi'”‘”‘s Online + HG out[ :]*([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,})/Total Regions : 1 Total Members : 2 World Estates : 3 Member Groups : 4 Avis In World : 5/;s/Unique Visitors[ :]*Active Users[ :]*Active Users[ :]*Active Users[ :]*Off Line IM'”‘”‘s[ :]*([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}11/19/2011[ :]*Last 30/D[ :]*Last 7/D[ :]*Last 24/H[ :]*In Queue/Unique Visitors 2011-11-19 : 1 Active Users Last 30D : 2 Active Users Last 7D : 3 Active Users Last 24H : 4 Off Line IMs In Queue : 5/’

  • Allan Carr

    Maria you have done a fine job! Thanks.

  • I wish you all the success with the site wherever you want to take it! You really deserve the attention it has gotten over the last few months. Kudos!

  • Drang Po

    can someone make available the php code so the rest of us can use the page? Seems the easiest way to create a standard.

    • I’ve also had a couple of people email me and ask for this — any developers out there?

      • tomorrow ill write up a php page that can display the results in json if requested. ie. stats.php?format=json
        it will be on my github and ill post a link here when done.
        Actually ill start on it now, got a test grid going that i can use

  • I still don’t have a machine-scrapable link for the stats for the following grids:

    AnSky, Cyberlandia, DWGrid, Marble Isle, My First Life, Neuland, RaWHiDe, Savage Grid, Speculoos, Virtual Worlds Grid, Your Alternative Life, CloneLife, Lost Paradise, Miki Kiti Tiki, Koeberle, Virtual-EPI, Gimisa, Anda World, VIBE: Virtual Islands for Better Education, AviWorlds, Sub Oceania, SLFDGrid, 3rd Life Grid, Admeja, Avalonia Estate, CHWGrid, iTEC, Bubblesz, MintaKa, Majickal Network, AweSim Worlds, Lost World, Majickal Life, Meet Instantly, Virtual World City, 2Open, Our Hometown, WestWorld, Bearly a Grid, Alluris Estates, Outworldz, YrGrid, Rock Grid, Baller Nation, Watcher’s World, Calypso 3D, Ventureworldz, and Kroatan Grid

    Here’s an example of the kind of thing I’m looking for:

    http://members.greatcanadiangrid.ca/GcgStats.php

    Current Time: 01-13-2017 00:19:07 UTC
    Total Land Area: 43515904 Sq. Meters
    Computed Total Regions: 664
    Actual Regions: 524
    Total Registered Users: 5184
    Unique visitors Last 60 Days: 1525
    Unique visitors Last 30 Days: 1059
    Online Now: 19

    Christopher Strachan has shared some code for generating something like this:
    https://github.com/chrisx84/opensim-stuff/blob/master/stats.php

    According to Kalasiddhi Grid, which just installed it, it works well and is easy to use. They say you just have to modify lines 2, 27, 28, and 29 for your own grid.

    Here’s their stats export: http://www.kalasiddhigrid.com/kalasiddhi_grid_stats.php

    The key things I’m looking for are:
    * Total land area (in either square meters, square kilometers, or computed standard region equivalents)
    * Total registered users
    * Active users, uniques for last 30 days, including both local and hypergrid visitors

    The less other stuff there is on the page, the faster it is for my database to pull it out (and the less burden on your servers).

    If you set it up, please let me know! Email me at [email protected].

    As part of the revamp, I’m also expanding the “most popular grids” list to feature more grids (to increase the number of links, and help folks find more grids to explore).

    So please let me know if you have a new stats page, and contact David or me with news items, new social media pages, land sales pages or prices, DMCA pages, TOS pages, currency updates, and new grid contacts.

    Thank you very much and have a happy 2017!

  • Kalasiddhi Grid just posted a how-to guide for how to install Christopher Strachan’s stats page code:

    http://kalasiddhigrid.com/guides/OpenSim/Adding_a_stats_page_to_your_grid_ver_1.0.pdf

    Thanks to Drang Po, the grid owner, for being so helpful!

    And Rich “Watcher” Williams made some updates to the code to make it a little easier for my database to scrape, which I also appreciate very much! Here is his code page:

    https://github.com/watcher64/OS_Simple_Stats

    Thanks everyone for working on this! Very much appreciated!

    • holy crap awesome 😀 ty so much kalasiddhi, sorry i didnt reply to ya yet, been busy with alot of scripting stuff.
      btw maria you can just call me Chris or call me the same name Terry Ford calls me which is Ven. My digiworldz avatar is VenumusVortex Resident.