E-Mail 'Upcoming changes to monthly stats report' To A Friend

20 Comments

1. 

   Good stuff Maria. Personally I think you have done a more than great job and would like to think that most appreciate the efforts you have put into this. Even the Know at all’s who always get anti re figures because they are too lazy or self important to contact you.

2. 

   Yep, right on. “A grid e byedMicrosoft or Facebook”, death by be. Being nice….. But hang in there, the metaverse is going to get interesting soon and you are sitting on your own rocketship. On stats, I would go further. As you said the grid’s benefit from the exposure, put it on them. Make them generate a stat file of your format. Kiss, but you design it,. Conversely for the small grid’s that don’t have the tech, or it doesn’t work for them at the moment, have a webpage that they can fill out. But if they don’t do either they get a no stats this month entry.

   An interesting thing would be to put the stats into a module into opensim. May be next version… Hint hint

3. 

   Thank you Maria for keeping the stats list going for all these years. I am certain most Opensim users appreciate it. I know myself just how time consuming and difficult it is to collect the data since, as you know, I built a grid search engine a few years back and I came up against all the issues you did. I also found I had to do a lot of work manually to try and get the numbers right and I honestly didn’t have time for that, running my own SL regions, Farworldz grid and the G+ Opensim Virtual community. So, I have to thank you for all your work and also Magnuz Binder for his grid stats list too.

   And Merry Christmas to you Maria and the whole Opensim community.

4. 

   Just a small correction: “Metropolis: Periods in the numbers where a comma should be”.

   Metropolis, being a mainly German grid, actually follows the original and general rule for separators in Arabic numbers, with a decimal comma and a point as thousands separator.

   It’s really just the UK, some of its former colonies, and a few countries heavily influenced by one of its former North-American colonies, at the time they adapted the Arabic numerical system, that use the decimal point and a comma as thousand separator.

   If either the English hadn’t stubbornly stuck to that “everything the French do is wrong” after England lost the Hundred Years’ War (1337-1453) to France (be it metric system, decimal system or thousands separator), or the American rebels had seen the light and adapted proper practice when the French won the American Revolutionary War against the British for them, the problem with _some_ North-Americans (actually, French Canada, Cuba, Haiti, Costa Rica and Greenland do it right) using the wrong separators wouldn’t exist. *winks at the American chauvinism about what “should” be*

   1. 

      as an American I bristle at your words!! (goes to look up bristle)…

   2. 

      I understand its a regional thing, but there are a lot of German grids out there — I’ve had to add the German words for “region,” “users,” and “actives” to my database (it’s “Regionen,” “Nutzer insgesamt,” and “Aktive Nutzer (letzte 30 Tage),” in case you’re interested.)

      Metropolis is the only one with the periods and commas switched.

      Some use no separators at all, some use the U.S. style, and many never get above 999 of anything so the issue doesn’t come up!

      What I can do is add a section of code to my database that checks if the grid is Metropolis, then strips out the excess period. That will slow down data collection — I’ll have to check every grid to see if it’s Metropolis or not. And it will mean that I’ll have to edit code over the holidays. But the again, I’ll have to edit code anyway, so I guess it’s no biggie. It would be nice if they fixed it, though, because it’s not just me — other people might want to get the stats to, either as a double-check on mine, or to put together their own lists (say, for hyperports) or for their own evil purposes (bwa ha ha ha).

      1. 

         “evil purposes” egadssss! but how about everyone else change theirs to how Metro reports and all will be just fine-))))))

         1. 

            I’d be all for the Lost Paradise HTML formatting (agreeing with Maria it is weird and requires a pretty complex sed to straighten out) with the Japan Open Grid stats labels (how about ハイパーグリッド アバター) and Metropolis number format (with proper decimal commas).

            1. 

               You’re a better programmer than me, Magnuz, if you can pull out the Lost Paradise data!

               (Of course, most people are better programmers than me!)

               For the Japanese grids, I actually don’t have a problem. In my database, I save the relevant keywords for each grid individually. So I go to the stats page and run it through Google Translate to see where the keywords are located. Then I copy-and-paste the original Japanese text into my database. I do the same for all the languages where I can’t just guess.

            2. 

               I guess I can do a general-purpose check… if the stat is supposed to be an integer, and instead there are three digits past the decimal point, multiply by 1,000….

            3. 

               Since one of the main aims with my stats is collecting concurrency figures, which requires a pretty high sampling rate, manual labor was never an option, but I needed to automate the sampling from the beginning, including finding ways to scrape even “strange” stats pages. So far, Lost Paradise and ATEK grid were the most challenging ones, but quite doable with my general approach.

               I don’t know exactly how you scrape the stats from the pages, but if it’s any help, I can describe my general approach, which at present leaves me with 12 out of 343 grids (3.5%) I need to use two lines each of special code to handle. Some of the cases are identical, so it’s just 8 special cases covering those 12 grids.

               I use shell script to strip the pages from linebreaks, HTML tags, extra whitespaces and thousand delimiters, then just grep for stats labels with trailing figures and strip the labels. A few pages I need to reformat before the stripping, a few afterwards, to keep the grepping standard and simple. I’ve tested the system with sampling up to 1800 grids every half hour, which is the sampling frequency I use, so the present 343 “active” grids don’t make it sweat much. Actually, it’s the TCP/IP stack rather than CPU or memory which sets the limit, so I probably could tweak it to handle even more.

               As an example, which might be of help then, the (Ubuntu shell) command I use to reformat Lost Paradise stats page into the standard format is (I wonder how Disqus will manage this one):

               sed ‘s/Total Regions[ :]*Total Members[ :]*World Estates[ :]*Member Groups[ :]*Avi'”‘”‘s Online + HG out[ :]*([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,})/Total Regions : 1 Total Members : 2 World Estates : 3 Member Groups : 4 Avis In World : 5/;s/Unique Visitors[ :]*Active Users[ :]*Active Users[ :]*Active Users[ :]*Off Line IM'”‘”‘s[ :]*([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}([0-9]{1,}) {1,}11/19/2011[ :]*Last 30/D[ :]*Last 7/D[ :]*Last 24/H[ :]*In Queue/Unique Visitors 2011-11-19 : 1 Active Users Last 30D : 2 Active Users Last 7D : 3 Active Users Last 24H : 4 Off Line IMs In Queue : 5/’

5. 

   Maria you have done a fine job! Thanks.

6. 

   I wish you all the success with the site wherever you want to take it! You really deserve the attention it has gotten over the last few months. Kudos!

7. 

   can someone make available the php code so the rest of us can use the page? Seems the easiest way to create a standard.

   1. 

      I’ve also had a couple of people email me and ask for this — any developers out there?

      1. 

         tomorrow ill write up a php page that can display the results in json if requested. ie. stats.php?format=json
         it will be on my github and ill post a link here when done.
         Actually ill start on it now, got a test grid going that i can use

8. 

   I still don’t have a machine-scrapable link for the stats for the following grids:

   AnSky, Cyberlandia, DWGrid, Marble Isle, My First Life, Neuland, RaWHiDe, Savage Grid, Speculoos, Virtual Worlds Grid, Your Alternative Life, CloneLife, Lost Paradise, Miki Kiti Tiki, Koeberle, Virtual-EPI, Gimisa, Anda World, VIBE: Virtual Islands for Better Education, AviWorlds, Sub Oceania, SLFDGrid, 3rd Life Grid, Admeja, Avalonia Estate, CHWGrid, iTEC, Bubblesz, MintaKa, Majickal Network, AweSim Worlds, Lost World, Majickal Life, Meet Instantly, Virtual World City, 2Open, Our Hometown, WestWorld, Bearly a Grid, Alluris Estates, Outworldz, YrGrid, Rock Grid, Baller Nation, Watcher’s World, Calypso 3D, Ventureworldz, and Kroatan Grid

   Here’s an example of the kind of thing I’m looking for:

   http://members.greatcanadiangrid.ca/GcgStats.php

   Current Time: 01-13-2017 00:19:07 UTC
   Total Land Area: 43515904 Sq. Meters
   Computed Total Regions: 664
   Actual Regions: 524
   Total Registered Users: 5184
   Unique visitors Last 60 Days: 1525
   Unique visitors Last 30 Days: 1059
   Online Now: 19

   Christopher Strachan has shared some code for generating something like this:
   https://github.com/chrisx84/opensim-stuff/blob/master/stats.php

   According to Kalasiddhi Grid, which just installed it, it works well and is easy to use. They say you just have to modify lines 2, 27, 28, and 29 for your own grid.

   Here’s their stats export: http://www.kalasiddhigrid.com/kalasiddhi_grid_stats.php

   The key things I’m looking for are:
   * Total land area (in either square meters, square kilometers, or computed standard region equivalents)
   * Total registered users
   * Active users, uniques for last 30 days, including both local and hypergrid visitors

   The less other stuff there is on the page, the faster it is for my database to pull it out (and the less burden on your servers).

   If you set it up, please let me know! Email me at maria@hypergridbusiness.com.

   As part of the revamp, I’m also expanding the “most popular grids” list to feature more grids (to increase the number of links, and help folks find more grids to explore).

   So please let me know if you have a new stats page, and contact David or me with news items, new social media pages, land sales pages or prices, DMCA pages, TOS pages, currency updates, and new grid contacts.

   Thank you very much and have a happy 2017!

   1. 

      maria you can get that same data in json format by just adding ?format=json to the url http://www.kalasiddhigrid.com/kalasiddhi_grid_stats.php?format=json buuuut i just realized that kalasid has HTML outside of the designated html area
      script updated with instructions as to WHERE to put html tags.

9. 

   Kalasiddhi Grid just posted a how-to guide for how to install Christopher Strachan’s stats page code:

   http://kalasiddhigrid.com/guides/OpenSim/Adding_a_stats_page_to_your_grid_ver_1.0.pdf

   Thanks to Drang Po, the grid owner, for being so helpful!

   And Rich “Watcher” Williams made some updates to the code to make it a little easier for my database to scrape, which I also appreciate very much! Here is his code page:

   https://github.com/watcher64/OS_Simple_Stats

   Thanks everyone for working on this! Very much appreciated!

   1. 

      holy crap awesome 😀 ty so much kalasiddhi, sorry i didnt reply to ya yet, been busy with alot of scripting stuff.
      btw maria you can just call me Chris or call me the same name Terry Ford calls me which is Ven. My digiworldz avatar is VenumusVortex Resident.