Two ideas for better search for OpenSim

Hypergrid SearchEverybody talks about the fact that we need better search functionality in OpenSim, but there are few solutions out there.

Why today’s automated searches don’t work

Keyword search is what Metaverse Ink does, created by hypergrid inventor Crista Lopes, also known as Diva Canto in-world. Searching for “shoes,” for example has two results — one on the Dune 2 regions of the University of the Aegean’s Dune grid, and one on the Serenity region of the Lost Paradise grid.

Talla Adam has a search engine on her Metaverse Traveller site. Searching for “shoes” in the “Hypergrid” category gives two results — the Bubblesz grid, and the CloneLife grid, but there’s no link to where you can actually find shoes on those grids.

The problem is — what do you search? Object names? Most objects have completely useless names, or are just called “primitive.” Same goes for region names and grid names.

A Google-style search isn’t an option right now. Google works by counting in-bound hyperlinks. We do have hyperlinks in OpenSim — they’re called hypergates. And there are hundreds of hypergates out there. The problem is, most of these hypergates are the multi-destination hypergates from The Hypergates — and they all go to the same destinations, 772 by current count. Plus, a few hundred gates aren’t enough for Google-style rankings. You need millions for really good results — and OpenSim is nowhere near that yet.

The other problem is that, right now at least, hypergates aren’t coded in any special way. They’re just an object with a script on it. And any object can be a hypergate. A shoe can be a hypergate. A brick wall can be a hypergate. A leaf hanging from a tree could be a hypergate.

So a Google-style search bot would have to crawl over a region and click on every single object in it to see whether it’s a gate or not.

Finally, what most people want to find is where is everyone hanging out? If I search for “shoes” I’d rather get sent to a shoe fair than someone’s shoe-filled closet, or, even worse, some spam region filled with plywood blocks all labeled “shoe.”

A Hyperica search for “shoes” gives 11 results — progress. Hyperica’s Avatar Clothing & Accessories category lists 74 different regions — which still a drop in the bucket given how many shoe stores there are out there on the hypergrid.

Plus, Hyperica is based on the principle of manual curation — like the early Yahoo! directory of the World Wide Web. Sure, some people submitted their destinations for inclusion, but most of the work was done by actual people sitting behind computers, surfing the Net.

For Hyperica, there is a form where people can add a new grid to the directory, and another form for adding a new region, but, so far, only a handful of people have used it — and all entries still have to be manually reviewed, categorized, ranked and added to the database.

First solution: A built-in events function linked to groups

Object, region, and grid names are rarely particularly informative. But group names and event names usually are.

My idea is to add event functionality to the groups function currently included with OpenSim.

So, let’s say Martha’s Shoe Emporium is having a shoe blow-out sale, the Shoe Lovers Collective is holdings its annual Shoe Faire, and My Favorite Grid is celebrating the grand opening of its Shoe Outlets Mall.

A search for “shoes” would return all of these, as well as any other shoe-related events marked public, ranked in the following order:

  • Events sponsored by groups I belong to would rank highest, followed by events with groups that my friends belong to.
  • Events sponsored by groups I have no connection to would be ranked last
  • Events happening very soon will be ranked higher and old events wouldn’t show up at all, unless I specifically searched for past events
  • Events sponsored by large, popular groups will be ranked higher, and events sponsored by small groups would be ranked lower

So, for example, if I was a member of the My Favorite Grid Events Group, I’d see the grand opening first. And if the Shoe Lovers Collective was a very large group, and their event was happening soon, I’d see that, as well. And if Martha’s Shoe Emporium Events Group had only one member — Martha — I’d probably not see her sale at all, unless it was the only shoe-related thing happening that day.

I would also want to have the ability to add an upcoming event to my personal calendar, and get reminders when the event was approaching. And I’d want to be able to invite my friends to an event I was going to.

One challenge would be how to get this to work across the hypergrid. You don’t want every grid querying every single other grid each time a user does an events search — that would be a nightmare.

One solution is to have a third-party service collect all this information from grids, organize it, then respond with the results.

Another, more peer-to-peer approach, is to query only those foreign grids where I already belong to a group — but that doesn’t help you much when you want to find new places to go.

Second solution: a new hyperlink functionality for OpenSim

This would work similar to the way that media-on-a-prim works, and would require the viewer and server developers to work together.

You would pick the surface of an object, and instead of setting a texture or a Web URL, you would set it to be a hyperlink. You would specify the destination, and either your own texture or a default swirling-water-hypergate animation. Plus, you would specify how the hyperlink is activated — by touching or by walking into it — and whether it would activate automatically — if the region’s threat levels allow — or only after a user agrees to the teleport.

You would also need to specify the link text, like “Click here to go to Martha’s Shoe Emporium.”

All this information would need to be stored in a database in such a way that it can be queried externally.

For example, right now the main external query is “get_grid_info” — for example, typing login.osgrid.org:80/get_grid_info into a Web browser results in a list of some key facts about OSgrid.

A similar query, “get_grid_links,” for example, could result in a list of hyperlinks, their destinations, and their descriptive link texts.

This would save a third-party search engine like Google from having to visit each and every region and test each object to see whether it was a hypergate or not. And would save grids from the load of these spiders.

Why we need this

As the hypergrid expands, being able to promote and discover new destinations will be key to attracting more and more people. In two years, there will be a bunch of new competitors out there, vying to be the platform for the new metaverse.

Having a working and usable search function would mean that we’re playing to win.

Do you have ideas for how to improve OpenSim? I’m gathering suggestions for a poll of Hypergrid Business readers. You haven’t added your suggestions yet in the comments of this article or in this Google Plus thread, you can add them below.

maria@hypergridbusiness.com'

Maria Korolov

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

  • hack13

    It is funny you talk about this, as we have been working on OpenSim.Directory for a while now and we are working to develop a system where it is enroll instead of crawling virtual worlds. The reason we don’t want to crawl virtual worlds, is new laws in Europe are making it harder for us to consider crawling virtual worlds as an option. We want to stay legal so an enroll system is what we are developing. I really encourage you to keep up to date what we are working on over at Zetamex, the directory is not just for grids, but for stores, merchants, and so much more. It is a directory system for all things OpenSim.

    • https://www.zetamextech.com/ Christopher Strachan

      also the web search part is a modified version of what i wrote for all grids to use as there own in world web search page that is up on my github for free under a open source license.
      The opensim.directory version will be abit different with a api system.

  • Fly Man

    The main issue with building a Search is how do you search places that are not Hypergrid based. For instance, if you go into OSGrid and search for something, it will come up with results of regions that no longer exist or are disconnected hypergrid wise.

    1 of the solutions that I have seen in the past months is a API that can be called on each region / grid that will return the results just like the normal search. The only difference, it points directly to the grid / region instead of saying “Hey, it’s like here”

    But the downside on this type of search is “Every grid would need to implement this API for it to work properly” meaning that it would need to be an open source solution so even the most basic grids can be handled.

    The last work I did was on a Lucene based search that handles all the regions within a grid and acts upon a question from a viewer. Secondlife has gone forward and is using the Google Search method for their web based search but most “older” viewers still use the basic search which is still based on the Lucene method.

    • http://www.hypergridbusiness.com/ Maria Korolov

      Fly — There is ALREADY a command in OpenSim that tells you whether a region is up or not. So if some regions are down, you can just skip those regions in your search results. Or rank regions that are up more often higher than other results (my preference, to avoid loading down their servers with too many query requests).

      If by “not hypergrid based” you mean closed grids — I’m REALLY not worried about this. Closed commercial grids can do their own searches any way they like — and they don’t have the scalability problem of searching the hypergrid.

      Tim — The problem with self-enrollment is that none of the good places bother to enroll. It seems to be a curse of human nature. Automatic scrapers and crawlers are probably the best bet right now, and from what you’re saying, it sounds like it should be a US-based company, with no European presence, that does this. Or, a company with a big legal budget to deal with all the compliance issues.

  • Talla Adam

    With regard to web crawling and European law I am no legal-eagle but from my reading it appears mainly concerned with the way in which data is being used, especially personal data. From what I read it is a matter of whether you can be classed as a data controller or not and that depends how you store and use data. If you are data mining as in the case of Facebook & Google with the aim of analyzing the data to build a personal profile on individuals in order to direct advertising on the web then you are a data controller according to European law. However, it appears you are not regarded as a data controller if your use of the data is passive in so far as you are just collecting metrics, snippets of information and web links to display. In my case, with Grid Search, the crawler will respect site owner’s meta tags anyway and it makes no attempt to read any content with the intention of building a profile on the site owner or anyone mentioned on it (it’s just not that clever anyway!). I did experiment with pinging splash pages, which are the pages the crawler looks at, to confirm the grid is online but that could be seen as a service attack so I dropped it. I also experimented with an lsl script to ping grids directly and report to Grid Search via HTTP and I dropped that too for similar reasons. Now I simply search myself and if a grid is offline a lot I close its entry in the database. So, I use both a mix of automatic crawling and personal moderation to keep the search engine database as uptodate as possible – generally, it is updated at least once a week but usually twice.

    For all new entries I don’t invite Grid owners to use an online editor but rather I personally look at a site and add keywords and a description. The description is basically my own opinion based on what I could find. If I notice a grid or standalone which makes a point of offering virtual goods such as shoes then I may add “shoes” as a keyword but, to be honest, I don’t have time to visit every grid to find out exactly what the owner might specialize in. I do find out things though from running Opensim Virtual on Google Plus and I make notes which may lead to changes in a grid’s search words later.

    My search engine is mainly concerned with finding full grids and standalone mini-grids to list and makes no mention of individual regions. I rather hope grid owners will do what Second Life does and offer links on their splash page to selected regions of interest. That is not to say I will never add a link to a selection of regions on a grid entry but it is not in my code yet. My code can accept searches for up to 20 search words but I may upgrade it soon to do more, like search the grid description too to broaden the search. But, anyway, the Grid Search address (www.x-retro-x.com/search.php) can be used in any viewer that has an internal browser and every grid found has a link to open the it’s splash page. In any event I would have preferred to see the viewer dev’s take this task on and give us a much better integrated search facility for Opensim worlds right their in the viewer prior to login. I made my search engine out of frustration with waiting so I included it on my blog as a useful Metaverse link along with other links I maintain to vendors, textures and resources generally.

    If anyone finds their grid is not found on my search then leave a message on the foot of the search page on my blog or contact me privately in G+. Thanks.

    • http://www.hypergridbusiness.com/ Maria Korolov

      Is there any way to get a list of the grids in your search engine?

      I’m also still having trouble figuring out how to use it. How do you decide what category to put grids in?

      Say I’m looking for the loginURI for “littlefield” — a top ten grid, so I assume its in the listings somewhere. I tried the hypergrid, general and community categories, then tried LFGrid. I never know when to stop searching — is it in a category I haven’t tried yet? Or is it just not listed yet?

      • Talla Adam

        Hey Maria. Try searching BDSM under Adult category. Any grid that is Hypergrid enabled should appear in that category as well but depending on the grids adult content it may not. For Littlefield the search words are currently: “erotic, BDSM, social, community, music, entertainment, sales, rentals, pony, girl, boy” but since this grid was entered really early in the search development I never added the grid name as search word so it wont be found using littlefield – I have added it now and it appears for Adult but not for hypergrid so, inspecting the algorithm, I find the code can filter out certain words to limit a search to Adult category (it works reasonably well up to a point but by no means that clever). I’m always open to suggestions of course so thanks for the mention. As for the raw data I have no made it possible to read that out from the database yet. So much to do, so few hours in a day – need, like, a thirty five hour day!