A new feature of Ubuntu was discussed today (which is like an announcement but without overhyping it), it is called Smart Scopes and is documented hereÂ https://wiki.ubuntu.com/SmartScopes1304SpecÂ go read that first and then I have a video for you to watch.

http://www.youtube.com/watch?feature=player_embedded&v=CBeQur7VBDM

Now go back and read the spec that I told you to read earlier, but all the way to the end this time.

In the video from left to right is Alan Bell (me), David CallÃ©, Jono Bacon, Michael Hall, Roberto Alsina and Stuart Langridge, all discussing this new framework for searching. It is coming soon, to the Ubuntu Raring desktop and then to phone and TV and tablet etc. The objective is to make searching really really effective and helpful to the user, but as with the previous efforts in this direction there will be some concerns around how it is implemented.

In short, Canonical will be running a server much like the existing productsearch.ubuntu.com server which will accept queries and return a bunch of results as json. The current implementation searches Amazon and the Ubuntu One music store and a few other places. The new one will do the same, plus more server-side searches, plus a new feature altogether which is a list of good scope names for the client to search. Your client will now send a list of all locally installed scopes to the server (actually a list of scopes you have added and a list of scopes you have removed or turned off from the standard set) along with your query. The server then returns results it found and wants to put in your dash, plus a subset of the local scopes you sent it, in order, that the server thinks would be good places to hunt for your search term. This means that your client might have 100 or more locally installed search scopes, but the server will advise it which are likely to give good results. Now for the scary bit, once you have looked at the results and perhaps clicked on something then your client pings the server again to tell it which scope produced the most relevant result. This means that the server can learn from this feedback about which scopes produce high quality results for that keyword, and perhaps rank that one a bit higher in future recommendations lists.

Lenses are now called master scopes

You control each individual scope that you want to search in or not search in, not the master scopes so you will have 100 or so things to turn on or off.

You can still have locally installed scopes that search authenticated data sources

You could in principal run your own search server if you write one to implement the API and patch the home master scope to look at your own server

The server isn’t open source

You can’t opt out of the feedback process (without turning off the smart scope altogether – which you can do)

If you install a local scope then your client will tell the server the name of that scope

Every query to the server is going to include a list of locally installed scope names (100 or so perhaps?)

You can focus a search at a particular scope by using a keyword, for example “omlet: chicken house” to only search the Omlet scope and not the chicken stuff master scope.

The rather poorly thought out remote-content-searchÂ checkbox to disable local scopes from doing online searches remains in place – however you don’t need it as you have per-scope controls.

There may be some code quality checks introduced to stop scopes that don’t pay attention to the remote-content-search settingÂ from getting into the Ubuntu distribution. – but you don’t need it.

This probably won’t put more adverts on your desktop while you are trying to do work.

This is probably a more private way of searching for stuff than googling for it.

This won’t be opt-in, all the good stuff in Ubuntu is turned on by default.

Your IP address gets logged on the web server logs, but not in the database of the smart scopes application running on the server. The developers working on the smart scopes don’t have access to the web server logs.

It would be relatively trivial (I could do it in a day or so if I felt like it) to write a gnome-shell client for this smart scopes server to display the remote results, however doing something with the scope recommendations list would be a bit of a struggle.

The home master scope (dash) search box will contain the help text “search your computer and online sources” to make it clear that it isn’t just a local search.

Now to the big question. How much are people going to freak out about this? Well if they read the spec all the way to the end they will see all the stuff that is being collected, how it is aggregated, how much or how little privacy this is costing them and why it is being done for the greater good of having decent search results. The feedback data collection process is likely to be slightly freakout causing. I can see why the developers want this turned on and I can see why it is antisocial to turn it off, like leeching on bittorrent while downloading an Ubuntu iso or whatever. I think they would be wise to have a checkbox in the privacy settings dialogue so that antisocial people can turn this off. I imagine the developers will stick with the current policy that if you want to use smart scopes you have to participate in the feedback process to make it better.

I think we need to do some education around the lack of an applications launcher though. Currently people think that Super + name of application is a replacement for the Gnome 2 applications menu. It isn’t. Super+a + name of application is how to start applications. This is going to focus the search on just applications and will work a lot faster than doing an omniglobaleverywhere search which is what the superkey does by itself.

For me this is a good development overall. The privacy debacle will be solved to my satisfaction when you can locally and personally blacklist scopes. This will mean that I can write a scope without it being co-dependent on all the other online scopes and I don’t have to worry about whether intranet access constitutes internet access. All scopes can simply stop ifÂ remote-content-search is set, but nobody needs to set it, the flag will basically just break all searching and be a bit pointless.