Beware the vanity metric

As the hypergrid grows, we should be wary of vanity metrics.

I’m a pretty big fan of the Lean Business model pioneered by Eric Reis of IMVU fame.  The main concept behind Lean is that you start small, release a barebones version of your product, measure carefully, and then modify the product based on feedback. Rinse and repeat until you succeed, give up, or decide things need to radically change.  Critical to that process is knowing what’s the right metric to measure. It’s very tempting to measure what Eric calls “vanity metrics” such as user registrations, or number of posts.

While these metrics sound impressive and make the creator — and investors — feel fuzzy inside, they are ultimately pretty useless metrics for figuring out how to make things better once things begin to go wrong. Metrics like “the average number of posts per user per day” are far more useful, because they let you tap into how your users or customers are actually using the product.

Hypergrid space

As the hypergrid and Opensim in general grows, I’m seeing an awful lot of vanity metrics being tossed around as solid indicators of healthy growth. There’s two in particular I want to call out:

  1. Region Count: It doesn’t matter how many regions there are in the hypergrid or any grid if they’re all abandoned 99.99 percent of the time.
  2. User Count: I know for a fact I have at least two different accounts on OSgrid I’ve abandoned for various reasons (passwords are hard). Maybe they still exist, maybe they don’t.  But one thing is for sure, they sure don’t represent a hyper-engaged user brimming with virtual social vitality.

I’ll give Active Users a pass, even though technically it’s pretty much a vanity metric.  At the very least it gives you a sense of how “active” — a pretty vague concept — a grid has been in the recent past.

This isn’t to say we shouldn’t share vanity metrics. They’re encouraging to look at, and give us a vague sense of what direction the wind is blowing in. However, when it comes to actually running our startups, we should learn to differentiate vanity metrics from useful ones and be honest with ourselves about which is which.

Now, having just poo-poo’ed several of the most popular metrics used to measure the growth of Opensim, it is only be fair to propose some other metrics to use instead.

These metrics are not necessarily for public consumption, but can be important for internal measurement of a grid’s health as a business.

These ones, while certainly not all that glamorous, would give us a better idea of what’s actually going on.

  • Average session length: How long are people actually staying on?
  • Average session region visit count: How many regions are users visiting?
  • New user login frequency: Are new users coming back, and after how long? Is there a fall-off rate?
  • Time-to first friend: How long do users go before they find someone to be friends with?
  • Text traffic breakdown: Which do users use more, region chat, private IM, or group IM?
  • Average download size per region: How much are region owners asking their users to download when they visit?
  • Average session friend encounters: How often, per session does a user end up within chat range of a friend?
  • Average population of teleport destinations:  Do users gravitate toward populated regions, or wander off to empty ones?
  • Time-to non-default avatar: How long does it take a new user to change their default appearance?
  • Region population turnover: How much of a region’s traffic is returning users versus new visitors?
  • Region percentage loaded: How much of a region ends up loading for a visitor during their stay?
  • Average Conversation Size: How many other users do users chat with?
  • Parcel Visit Distribution:  What parcel do region visitors end up visiting the most? For how long?

There’s probably hundreds of other questions we can ask. It all depends on what we’re trying to find out. The important thing is to ask in the first place, and then set out to try and answer them.

Region counts on the 40 largest public OpenSim grids -- but size does not indicate success.

Region counts on the 40 largest public OpenSim grids — but size does not indicate success.

Getting these metrics will probably require the collaboration of grid operators, region owners, and in some cases viewer developers.

What metrics would you like to see the data on?

(This article adapted from a post at MetaPundit and its follow-up article.)

nexusomega@gmail.com'

Alan Tupper

lan Tupper is an Open Metaverse evangelist, technologist, and writes the blog MetaPundit. He also draws occasionally and is the co-creator of Kamikaze.

  • Gaga

    Good article but I wonder if this is going the path of Facebook style privacy infringement. I mean, counting how many times individuals speak to each other? And monitoring people’s teleport destinations is a bit like snooping – not good in a free Metaverse, surly?

    In any event no matter what metric gets displayed they can always be gamed in an open source world. Measuring the growth of the Metaverse in general though I think the grid and standalone count is a good metric since at least one individual and possibly friends and supporters are working to maintaining each instance.

    • AlanTupper

      Privacy absolutely is and should be paramount when someone sets out to get these metrics. I agree that there’s a very fine line that needs to be tread. Facebook-style data harvesting through fiat is toxic, and in my opinion harmful to an open web. That being said, it’s really hard to make meaningful changes when you don’t have any way of objectively measuring differences in behavior.

      Perhaps the way to go about it is to modify the viewers to contain a configurable list of potential metrics for users to opt in to. It would be a good place to fully describe to the users what’s being collected, and would (in theory anyways) put the discretionary power in the hands of the users. If a user felt truly uncomfortable with any metric tracking, then they shouldn’t be obligated to give anything up. Any metric data collected should be thoroughly anonymized, regardless of a user’s preference over individual metrics. The point is to provide statistical data points, not snoop into a user’s personal affairs.

      You are absolutely correct that metrics can always be gamed, and even the metrics I called out have meaning. Ultimately it boils down to how much we as a community invested in the growth of a open Metaverse are willing to share when it comes to actual user behavior, and how we use that information to refine the experience. Without metrics, we’re pretty much just making guesses.

      • Gaga

        Yes, I do agree there should be a configurable list of potential metrics for users to opt in to if data is to be collected for analyses and provided those concerns and user privacy are seriously address then I have no problem with it.

  • Danko Whitfield

    I think Alan makes good points here and as owner of a small (personal) grid, I’d like to have some of the measurements he suggests…but I do think region count and active users are useful figures that tell us something. I agree that Total Users is a totally useless number. I wish it wasn’t but it is. I’d rather see Active Users Last 12 Months.

  • John Oeffinger

    Agree that we need to have better analytics while respecting our customer’s privacy.

    Alan provides some useful measures, and some are more focused on which technology component is used and how often.

    I think the folks at Metaverse-Business http://www.metaverse-business.com/ have a solid approach. We used their region array counters in our SL sims. If you look at their reports, they provide information on Visitors, Locations, Avatars, and Countries. They provide three different heat maps for location information. The Avatar information breaks out the SL age, payment status, and language among other metrics. Visitor information gives you information on returning avatars, length of time, hour of the day, etc. These measures give us very useful information. In our case, we were able to track our progress and identify opportunities based on the analytics they provided.

    Alan’s measures also assume a 24/7/365 grid. I believe we need to look past this concept. These full time comparative grid conversations are very similar to the discussions we had in measuring SIG or Forum growth in the early days of CI$ and GEnie. The problem is they do not get to the information you really need to make business decisions.

    Today’s technology also enables us to unbundle services. In 2000 we were talking about learning objects in eLearning. LO’s were supposed to be the building blocks we could use for mass producing eLearning courses. While LO’s never achieved the success envisioned by their creators, we did see content creation unbundled on a large scale. The content creators in SL create an enormous amount of virtual goods. This is nothing more than unbundled content.

    We can take unbundled content a step further with grids, worlds, regions.

    Today, we can talk about on-demand immersive environments. Kitely’s business model is to unwrap and display the world or region only when it’s needed. This is a very effective and efficient way to provide on-demand opportunities, especially for learning and other event driven models. Why keep a world online 24/7/365 when it is only needed a few hours a week, or for a one-time event every week or month?

    Most of the metrics Alan suggests wouldn’t be effective in an on-demand world. On the other hand, the Metaverse-Business analytics would apply to both on-demand worlds and 24/7/365 grids.

    I believe there are roles for both types of models, on-demand and 24/7/365. The key will be to find analytics that work in both models without favoring one or the other. In this way we would get a better handle on what is truly happening.

    I’m glad Alan created this post. It reinforces the need to develop solid analytics because the vanity metrics may not be what we need.

  • Though, of course, I agree that ANY numbers can be gamed, and are in several regards, I would very much like to see these two.

    “Average session length: How long are people actually staying on?

    Average session region visit count: How many regions are users visiting?”

    Or at the very least, the first one.

    otoh, grids, especially commercial grids, being extremely competitive, some negatively so, are not going to be much interested in displaying numbers “for all to see” that are transparent…I think they are doing an exercise in “wrongthink”, but it’s their business, not mine.

    But I suppose some of them do internal number counting in some of these regards, and others, that will never see the light of day to their customers, and potential customers.

    Nothing intrinsically wrong with this, of course, as businesses do not share their P&L statements, as a rule, to their customers anyways, so why share other numbers.

    My issue has always been in how some numbers are used, and gamed, in a bit of a disinformation/misinformation type of way. Often, but not always, actually not by the grid owners themselves, but their grid lovers/promoters who often have a self-interest motive.

    • One thing I think would provide a better picture of grid usage is to use a histogram of average usage patterns instead of a single average, for example: how many sessions were 0-5 minutes, 5-10 minutes, 10-29 minutes, 30-59 minutes, 1-2 hours, 2-3 hours, 3+ hours.

      Another useful histogram would be to count the avatars spending these times inworld and not the sessions themselves.

      • I totally agree. Personally, I would love to see grids put numbers up in different ways than the standard ones…if anything, it would make their splash pages much more interesting to look at.

      • hi Ilan, is it possible (in an easy way) for you to export my history in a format i could blog about? i started making a spreadsheet a few months ago for the time spent in each of my publicly viewable worlds

        i don’t want people’s names, only total time to date for Hax Nuit, the freebie store, cool ice campus, and space base

        my purpose is to encourage others to share their work, even if it is as a freebie sim in Kitely in the hopes that the more that others see, the more likely they will find something they like and give it a try themselves

        thanks!

        • Hi Ener,

          Are you asking about getting what you see in your History page or just the total time people who aren’t your avatar have spent inside each of your Kitely worlds? (everything is possible but it will take time to extract that data and we’re very busy working on several high priority tasks at this time)

          • no worries Ilan, i can catch up on my spreadsheet =)

  • Merrie Schonbach

    Im so glad you wrote this article, this is also something that I have noticed and glad you offered some really good alternatives to look at. (AKA Andress Renault)

  • “It doesn’t matter how many regions there are in the hypergrid or any grid if they’re all abandoned 99.99 percent of the time.”

    well, maybe we don’t know if that is important or not? who is the person that can absolutely say this is a true statement?

    for more than 99.999% of my life I have not had a polio shot . . . but the one I had was arguably important

    same with air and water, i only use a tiny bit but there are benefits to having an excess that is unused at any one time

    i guess my point is to be careful of using absolutes. there is a value to knowing that 99.99% of regions are abandoned and that is data that can be acted upon (albeit, we also thought burning fossil fuels was a totally awesome thing starting in the industrial revolution – limitless and inconsequential)

    excellent article and one that provoked me into thought – which is an awesome goal of worthwhile writing! =)

    • I’m with you on this, Ener.

      And some personal thoughts: Back when I first started doing this, MOST regions I visited were bare grass. Sometimes a few prims lying around, sometimes a construction site. The built-up regions were clearly built by techies — all cold and futuristic, not really designed for human occupancy.

      Today, when I visit a region — unless it’s just recently been set up — it’s usually a place worth visiting. Even regions that are mostly landscape are designed landscape, with rolling hills and field grasses and wetlands.

      Whether they’re empty or not depends on two factors:

      * Is it a welcome area or shopping area on a large grid? Then there are usually some people there.

      * If there an event going on? In which case yes, there are people!

      But you might say, if you randomly drop into a random region on a random grid, it’s probably going to be empty. This is true.

      * Residential regions are only busy when the owner is on them. Since the owner only spends a percentage of his or her time online — and only spends part of THAT time at home, and the rest on other regions — you’re probably not going to meet the owner.

      * Roleplaying regions, bars, concert halls, classrooms and other venues are typically only busy during scheduled events. The rest of the time, even if someone does drop by, they see that nobody is there, and they leave. Roleplaying and bar-sitting are not fun alone. Events might happen once a week, or once a month, or even once a year. OpenSim regions are cheap enough that people will leave them up, just in case they need them.

      * Lag and bandwidth constraints mean that people would rather spread out then concentrate a lot of activity into one small area. Reducing population density and prim counts results in better performance and a better experience for users. In Second Life, there’s an artificial constraint of land scarcity due to the high price of regions that forces people to build more densely than they might have otherwise. There’s no such constraint in OpenSim.

      Personally, I don’t see empty regions as a problem for OpenSim, but a business opportunity for someone who figures out a way to help people find out where stuff is happening.

      Technies are still stuck on the idea that we search for information — where are objects named “chair” located, which groups have “sci-fi” in the title, which users have an interest in virtual golf. These are old school searches, key-word searches, pre-Google type searches.

      In the metaverse, what I want to know is:

      * Are all my friends hanging out somewhere having fun without me?

      * Where are the fun parties at?

      * I’d like to go to a business networking event tonight. Are there any scheduled?

      * I want to learn something. Are there any classes starting soon?

      * I want to go shopping. What are the popular malls?

      A couple of years ago, Nielsen was tracking the number of minutes people spent in games each week, with Second Life in the top 10, ranging between 500 and 800 minutes a week. Nielsen stopped tracking these numbers, but let’s say that OpenSim would have been on the lower end of that, at 250 minutes a week (since many OpenSim users also split their time with Second Life).

      As of mid-October, OpenSim public grids had 30,988 regions and 21,189 active users. So, on AVERAGE, each region would see … 170 minutes of use a week, or a little less than three hours… or about 1.7 percent. So, if people were evenly distributed, the average region would be empty 98 percent of the time.

      Since people cluster together, the percent empty will be even higher.

      On the other hand, 99% of earth’s land area is also devoid of human population: http://wiki.answers.com/Q/What_percentage_of_land_on_earth_is_dominated_by_humans

  • Revel Peters

    I have about 5 accounts on osgrid not because i wanted them but because due to database issues over time they became inaccessible and I would put in bug reports but to this day they remain unusable. There are many people like this. Total accounts on something free is useless. If this was a paid service then total paid accounts would mean something but users just passing buy taking a 5 minute look and leaving or users like me who have had to create new accounts over time due to grid issues are not really things that mean anything 🙂 they mean what they mean and have nothing to do with user engagement 🙂

    • Joe Builder

      Seems to me you have a serious issue with your PC, Never really heard that before having to make so many accounts because of a asset issue.

      • Revel Peters

        my pc? this is a database issues osdgrids issue and its not always the same pc so unsure what that means? they hold the avatar info not my pc i just use it to login using a third party browser

        • Joe Builder

          Didn’t know osgrid had database issues we talking past or present? If its past issues that’s not relevant to today. And also if your assets are corrupted doesn’t mean a new account is in order.

          • Revel Peters

            I was responding to the topic not about past or present or whether its relevant today or not. Stop being so emotional or defensive and step back a bit. I dont have assets it was a character I made to login to the grid with period end of sentance. And yes those characters still do exist and to this day are not usable. I wont be responding again.

          • Joe Builder

            Just trying to decipher you comment is all, Being it makes no sense. Seems you have other motives that pertain to Osgrid.

          • Revel Peters

            um no there are no motives. I commented on the use of metrics and gave an example that came to mind its the largest grid and uses this metric. Period end of sentance.

    • Arielle

      Usually best to go to the irc #osgrid chat and ask there to have your inventory be reloaded from a backup. http://www.osgrid.org/index.php/freenode There were a couple times in the past few years where that was necessary for many of the residents.
      Joe is pretty new so he would have no idea of that. 🙂

      • Joe Builder

        LOL pretty new, If you only knew :)~

      • Revel Peters

        yeah not a big deal really it was simply a comment I made in relation to the topic I almost regret saying anything because something is being made out of nothing lol. They were nekid avatars really and I like many just say “oh well” and create another one so the stats get inflated. All databases get scrambled from time to time I just did not care enough to have my nekid avatars fixed. I had one repaired but the rest still float around …… 🙂

  • Arielle

    A couple of metrics that noone ever mentions and yet I believe are indispensable gauges to to the health and vitality of Opensim as a whole are a) the populations and chatter of the Opensim-dev, Opensim and to a lesser degree the Osgrid Irc channels. And b) the amount of new revisions put out monthly of Opensim Master code. The populations of the irc channels are dropping and the chatter has subsided to occasional unanswered pings and questions. This to me says a lot about the long term prospects of Opensim as much as I hate to point that out.

    Though I have not tried to determine the rate of revisions to Master code month to month, I suspect that it has dropped significantly also. Hopefully just temporary blips.

    • Hannah

      I agree that those are signifigant, but not necessarily right now. It’s the holiday season, and things tend to taper off then anway.

      As far as tracking revions, couldn’t you use the ohloh stats to track that? http://www.ohloh.net/p/opensimulator?p=OpenSim

      I think you’d have to watch those over time, say -six months or so, in order to get any feel for wether they’re increasing, flat-lining or decreasing.

      It’s my opinion, though that while development (AND maintence, as well as security troubleshooting) is desireable, if 0.7.6 was the last official release it would prove stable enough to sustain the opensim ecosystem for a while -at least until LL changed the SL protocol enough to make TPVs inefficient at communicating with opensim.

      • Arielle

        Thanks for the link Hannah, I had forgotten about that site. It shows that the last few months the development process has suffered a big decline. Possibly a hangover of the work pre conference.

        The Irc channels however I have been watching since I started lurking there back in 2010 and can safely say they have dropped significantly since that time. One could argue that Opensim is now represented in other mediums like G+ and here on HypergridBusiness but I would argue that there isn’t the technical discussions like there was on the Irc channels in past. Possibly a result of more people moving to hosted and closed grid solutions where one can leave the techy stuff to others.