Joho the Blog
An Entry from the Archives

« Wish list: Wait for Message || Back to Blog | Pray for Reason »

June 20, 2004

Semantic Behavior Index

Jon Udell speculates on what our OS would do if Google wrote it instead of Microsoft:

On the Google PC, you wouldn’t need third-party add-ons to index and search your local files, e-mail, and instant messages. It would just happen. The voracious spider wouldn’t stop there, though. The next piece of low-hanging fruit would be the Web pages you visit. These too would be stored, indexed, and made searchable. More ambitiously, the spider would record all your screen activity along with the underlying event streams. ...

Interesting idea! And couldn't we implement enough of this to test its usefulness pretty quickly? After all, macro programs such as ActiveWords already watch our every click and stroke. I believe ActiveWords already keeps a history. Of course, that wouldn't tell us the precise state of, say, the word processing document when we jumped over to our browser window and typed in an URL, but it might still be useful.

When I say "might," I mean it. I'm not at all sure I'd actually use such a system. It might feel invasive and it might have to operate at such a low level that it introduces deep-seated instability. More worrisome, I tend to be so distracted in my work patterns that the sequence of my small-motor movements may not be a good way of searching for the threads of activity.

Behavior obviously contains clues about the intent that stitches actions into meaningful streams, although the clues can be awfully misleading: If you see that I move from a web page to a word processing document, there's a chance the first inspired me to write something in the document, although it's also possible that I got bored reading the Web page and decided to get back to work. If I copy from the Web page and paste into the document, you have a stronger clue.

A Semantic Behavior Index could be better at inferring third-party intent from behavior than we humans are, although it's hard to see it getting better at interpreting my behavior than I am. But it doesn't have to be that good to be useful. The question is, would it be useful to have a searchable table like the following:

TIME

WINDOW

ACTION

CONTENT

10:23:13

"Chapter1.doc"

Typing

And so we see I was right all along

10:23:14

Desktop

Opened browser

C:\Program Files\Mozilla\FireBird.exe

10:23:18

FireBird.exe

Typed in url

"www.wikipedia.org"

10:23:21

http://en.wikipedia.org/ wiki/Voynich_Manuscript

Typed in search form

"Voynich"

10:23:30

http://en.wikipedia.org/ wiki/Voynich_Manuscript

Copied text

"Over its recorded existence, the VMs has been the object of intense study by many professional and amateur cryptographers "

Would something roughly like this be feasible? Worth indexing? Would it be useful without that fourth column, since that'll take up a lot of HD space? Or would it all be nothing more than noise and an invitation to come invade our privacy?

Posted by D. Weinberger at June 20, 2004 11:41 AM


TrackBack

Listed below are links to weblogs that reference Semantic Behavior Index:

» http://80.177.26.60/blojsom/blog/cubicgarden/xml/Semantic web/?permalink=F713A85C2618F83F577AE13CBD82D69E.txt from
[Read More]

Tracked on June 21, 2004 06:43 AM

» Google is not PC from the iCite net development blog
In response, David suggests a somewhat different approach. . . it isn't about the way Google looks at information as much as the way Google might look at the information tied to the legacy software and operating system structures of your PC. [Read More]

Tracked on June 21, 2004 02:14 PM

» http://WWW.tsuibhne.net/knot/archives/000281.html from Life As A Knot
Joho the Blog: Semantic Behavior Index Jon Udell speculates on what our OS would do if Google wrote it instead of Microsoft:... [Read More]

Tracked on June 21, 2004 06:42 PM

» Semantic Behaviour - Getting Somewhere from A Networked World
I think Dave Weinberger has unintentionally gone off on a tangent with an interesting post looking at what he calls a Semantic Behaviour Index, following Jon Udells' thoughts about a Google desktop. Dave quite rightly gets nervous about privacy issues [Read More]

Tracked on July 5, 2004 03:11 AM

» Google Desktop Search Proving Popular from mediajunk
Google has trumped Microsoft by releasing its new desktop search tool, which allows you to search documents on your own... [Read More]

Tracked on October 21, 2004 09:56 AM

Comments

I'm interested in indexes / lists like this too, and, though I like the idea of being able to play back certain things I did in the past, I really like the idea of allowing different people to "play" public versions of each others' indexes / lists.

(I think indexing collaborative / group behavior is way more interesting than indexing one's own.)

For my iCite net experiments, I am working with a small set of "semantic behaviors" that I can index (and publish) with a single-click, e.g., read, not-read, filed/archived, and (especially) cited. Automatic indexing would be nice for some things, but I think there are some interesting things to explore specifically with user interaction with the list / index.

Posted by: Jay Fienberg | June 21, 2004 01:59 AM


In the last instance you make reference to invading privicy.

This idea running as a OS would be worry-some. Running as a client application with good controls and a experienced user would be ideal. IMO something like the zonealarm popup box system would be useful so you could teach the client app what content can or behaviors can be logged? I know it would be very ignoying at first but soon things would settle down.

The other thing I thought about, if the client app watched logs and caches. So for example my jabber client creates a log of all my communication which I choose to delete or keep. My browsers create cookies, histories and cached content based on what I do. I wouldnt object to a client application which read this information and made a semantic behavior log.

In the whole its an interesting idea and just like Jay, I would love to see a public play.

Posted by: Ian Forrester | June 21, 2004 06:34 AM


Ethan Zuckerman sent me email from Ghana that this is something like Gelertner's LifeStreams work. Ethan points to http://www.wired.com/wired/archive/5.02/fflifestreams.html?person=ted_kaczynski&topic_set=wiredpeople

Posted by: David Weinberger | June 21, 2004 07:49 AM


We, i.e. ActiveWords, don't keep a history. We have always been cognizant of the privacy issue. We recognized that this was such a sensitive issue that any breach of it would have been fatal to our evangelizing our product.

With ActiveWords, every time you trigger an ActiveWord, we clear the buffer, and provide you with a clean slate so to speak.

I am familiar with Gelertner's work on a limited basis, and I personally like a lot of what he has done and written about.

I see a huge value to the idea that would allow me to point or search easily data in a continuous stream. An example might be that let's say I treated someone to lunch at a Chinese restaurant. I would probably post the expense to Quicken, and would make notes about any follow on items. At a later date if my friend E-Mailed me as to the name of the restaurant, I shouldn't have to puzzle through various applications to figure out what the name of the restaurant was, but rather just to search or point to the character string to find it, and in so doing see the context. It might prompt me to follow up on other things.

It seems to me that real issue is encryption. Hard drive space is getting real cheap, processor power more than adequate to accomplish these tasks. We have noted that the faster computers run, the better ActiveWords works. The lag in productivity comes from human factors, the “I know it’s here somewhere, just don’t know where syndrome!”

Notwithstanding the heat that it takes, I see people using Outlook as a file cabinet, and now with applications like Lookout, you can find stuff pretty quickly. It is not whether people do this kind of stuff but how.

Along what I call the ActiveWords Odyssey, I met a very bright woman who suggested that we had done a good job of implementing the basic communications metaphor. I didn't know what she meant, and she explained that we as humans have impulses, followed by utterances seeking predictable results. She argued this idea is fundamental, that it doesn't change, and that it won't change.

Hence if Jon's ideas are implemented, and if implemented by Google, they will merely manifest what we all want.

Again, in my mind it goes back to encryption, and how do we protect this stream of text, not whether we want or will use it.


Posted by: Buzz Bruggeman | June 21, 2004 10:10 AM


Stop giving DARPA more ideas! Or worse, a possible architecture for CAPPS III! :)

Posted by: Mark | June 24, 2004 03:11 AM


Hello,
I fully appreciate all the comments. According to me Semantic behaviour index attract the people. I familir with little bit. I really like the idea of allowing different people to play public version of each other's index/list.

Posted by: William | September 20, 2005 04:33 AM


Post a comment

Guidelines for Commenting

Basically, you can say what you want. (Click here for the fine print.)

If you haven't left a comment here before, your comment may be put into a queue for me to approve. Sorry for the delay. Blame the damn spammers.