-
Website
http://www.scobleizer.com/ -
Original page
http://scobleizer.com/2009/06/28/real-time-systems-hurting-long-term-knowledge/ -
Subscribe
All Comments -
Community
-
-
Top Commenters
-
Popular Threads
All Comments
Welcome back.
You are validating my "Noise to Knowledge" idea which I noticed in the mid 1990's as we started to get computing system ubiquitously available to people. New technology was arriving (email, internet, word processing, spreadsheets) and old technology was being gradually usurped (newspapers, magazines, books, libraries, published standards, etc.).
The idea is that information goes through three steps: Noise to Best Practice to Knowledge. Further there is flow between all three states. In the old days before computers we learned how to manage flows from Noise to Knowledge. For example, we were brainstorming near the water cooler and thought up some neat ideas (Noise). After returning to desk we tried out the best (or those we could remember) and say, "Yea, this works". That becomes a Best Practice. We start telling others and it gets to be the way everyone does it. Knowledge. Eventually, the Knowledge needs to be refreshed and we go back to brainstorming for new ideas or look to current Best Practices to change knowledge.
Protocols for doing that before computing were understood, in general. After having computers, we evolved into becoming tool-centric. For example, email became (and maybe still is) not only our source of noise (daily mail) but also our source of best practice and knowledge (people individually storing old mail in folders for reference). This is bad as it's not scalable. But if a good email program is all you have, e.g. Outlook, then that's what you're going to use.
I think the trick is to have great tools fit for purpose and tools for Noise are not the tools for Knowledge. Twitter is Noise. Google searching is probably a lot of noise, but Best Practice can easily be found. Knowledge via Google can be discerned but not easy and there will be many versions of the truth.
I wrote all this up into a paper at the time for discussion with the IT Department to demonstrate why the tools they gave us were inadequate and will be mis-used. They were. And the IT Dept didn't change their direction. At the time intranet web servers (Apache) were very suspect and declared "non-strategic", as was the Internet (!). Oh well.
I re-wrote the paper last year and posted on my blog.
But that insinuates that you logged out of those and are no longer addicted. I would say both are false.
Following the gist of your argument, it is well established that Twitter's search is very broken. I hear rumblings about them improving, and maybe they will go back and get it all, but I have concerns they may never get it. There is a place for both real-time and long-time discussions being made available where they best belong.
http://tweetnews.me
I really like this model and am really surprised this hasn't become the norm or taken over search.twitter.com
I just don't see my mom or dad searching Twitter directly as the results are noisy and unreliable
But with this model it adjusts to their normal setting.
I think the tacit knowledge that you have gained using real time has been converted into explicit through slow time blogging, daily actions.
I say real time conversations are more like oral conversations. Which Knowledge does not have to be written to be maintained and passed on. Ie. Navajo language wasn't written until 50 years ago. But they still have lots of knowledge from long ago.
It is easy to get to the conversational real time knowledge, doesn't sound like it and I am glad that you have pointed this out. seems like a business opportunity for some one who can thread archive and than provide search at a later time......
Of course that won't solve Robert's wish for Time-bound snap-shots that can be easily resurfaced: Google itself, even with the recent "Recency Operator" improvements (Last 24hrs/week/month/year), still has a way to go before you can say "give me date range from: ... until: ..."
Sound to me like there are plenty of business opportunities around providing archiving abilities around given real-time queries. I.e. Robert can find what he wants on a given day or recent sequence of days SHORTLY AFTER the event, and then says: "Hoover up" all of this info and archive it for later... e.g. for the recent "140conf" or "140tc" etc.
That implies taking timely action because Twitter seems in no mood to let you back-search past about 30 days (at best, often it's only about 7 days anymore during daytime loads). They may of course already be selling access to the full range to corp. researchers for a lot of money.
While I agree with you about the poor search functionality available to Twitter (not being a user of FF services) - that you cannot filter your searches to dates, themes, persons etc let alone use of locational information (which tbh is more limited by the means with which people use either service) that is not to say that it's a problem with RT services.
Should there be an improvement from the assumption of time-series based data (which is what the services are) away from sorting descending by date posted then there may be some improvements there. The fact that Twitter (and FF?) offers private datastreams complicates the matter greatly (eg: how many facebook indices can google search when the data is kept completely behind the garden wall?).
I think there's a lot more that could be done - flickr's current search method options is a good example - and with time-series data that is immense in size and value (but with little "rank" or "peer review" and linking obfuscated through the use of shortened urls) the value of those searches (both generally for APIs and visualisation of trends and for specific users where private streams exist) would prove a great benefit to the company.
Twitter, Friendfeed etc. are like chatting and discussing at the office, like doing workshops, but with everyone listening who would like to. Blogs are like putting together what you've learned with what you've made up by your own and pinning it down.
Everyone who has ever written a paper for publication or a serious blog posting knows that it is a big difference to think that you're into it after chatting, and to really pinning it down in a well structured piece of text. So, I guess, blogging it is more than just making knowledge retrievable. It is making it explicit and shining in the first place.
So, Scobleizer, please stick to it!
I listen to music on Pandora but buy it on iTunes if I want to be able to listen to it at my leisure. Can't the real time internet and the regular internet co-exist in the same way?
-chris
Every problem is an opportunity for someone else. One could bring the most interesting tweets or most important of them, into one's blog and tag them so they are searchable not only for themselves, but also on Google and get some google juice to boot.
Granted, there are tools that will make Twitter search more useful, but we still have the issue of context and historically structure.
I would say that with the new search tools released by Twitter (timeline, wonder-wheel, context search) we are getting closer to where we should be. Not near it, but closer. Alas, the problem is that the next step (a truly semantic web) is too far for most people to grasp today -- and thus, Twitter and friends continue to gain adopters.
If you are "part of the conversation" in Twitter, you MAY learn something, if not it is more or less gone, like any other conversation anywhere.
If you want to share knowledge with people, Twitter is probably the worst tool. If you want to inform people about the latest happenings in your life, or ask a "random" bunch of people a question, Twitter MAY be useful.
Go buy the bike with the most suggestions? That will get you the most popular bike, but not necessarily the best one for you. This happens with camera recommendations all the time: a prosumer DSLR is not the best choice for many questioners, but its what the camera enthusiasts use.
Investigate every suggested model? Possible I suppose, but not very practical.
Facebook can improve this situation somewhat, in that your social graph should eventually allow you to express whose opinion you trust more than others. So you'd have a weighting based on who made the suggestion. Nonetheless this is still imperfect, it isn't realistic to have a single trust metric for every possible subject.
What we lack is a measure of authoritativeness of the _answer_: both an inherent quality of the person plus an indication of how sure they are of the answer they are giving. Suggesting a bike based on their own experience of test-riding a dozen models six months ago is far more authoritative than suggesting the cool bike they saw on University Avenue yesterday. There is also a timeliness function: test-riding bikes 6 years ago is less useful than 6 months.
How to generate this trust metric? Requiring people to do anything manually, to rate the strength of their answer, is doomed to fail. It is both annoying and easy to game. Measuring how many recent events in their lifestream are relevant to the question might be one way. If they mention their timing to bike to work each morning, they probably have much more interest in and knowledge of bicycles than most.
Then again, the person who bikes every day isn't going to be the best source if you're only planning to use it twice a month. They're likely to suggest "too much bike". You can still see how access to the lifestream would help though, by gauging how closely their activity in this area matches yours.
The twitter knowledge, the blog knowledge it is all very cumbersome. Blogs in particular haven't really changes since um early 80's. Its the same thing, dial in on my dads 9600 baud modem to the local bulletin board, yeah maybe you had to wait 30 minutes. Read the message type a message.
How far back can we retrieve those conversations? Now I would bet, that everything that has ever been twitted is stored some where. Could be wrong, but it would not make sense for twitter to destroy their content. It would kind of being like burning money.
Back to basic blogging and such, the problem is that the original poster says something. People comment and yes if you scroll through and read them you can parse some Knowledge from them. But why doesnt someone have a closing space on a blog. The keypoint is this the thought leaders post and let loose and move on to the next, it is so fleeting. If they were truly interested they would close the gate once they opened it. They would analyze the comment contents and they would deliver the meaning of what they learned. How they changed from the shared distributed knowledge.
That is the creative process, but blogging is mostly about posting something, letting the trolls take it and moving on to the next. yeah it is retrievable but is it usable?
There is an old saying "Don't put all your eggs in one basket."
With all things human, these pearls of wisdom can be broadly applied. As with our virtual interactions with each other we must balance the use of all technologies.
As you have documented so many times, instant communications such as Twitter and Friendfeed have great advantages. But as you bring up there are restrictions or draw-backs of these technologies.
Could you imagine trying to catalog, quantify and qualify just your own Tweet/Friendfeed "though channel" over a period of 10 years?
Just the massive amount of data would be overwhelming.
So, I guess if there are thoughts you want to preserve for historic reference you might want to sit down and give those thoughts the time and effort they deserve to be preserved.
Although chatter may lead to knowledge and action as demonstrated with the recent Irainian elections, when you have a lot of chatter you have a lot of noise as well and the good thoughts most likely will get drowned out and not archived very well.
One problem with our "gotta get it now" and "24 hour news cycle" is that people really don't take the long view of humanity and knowledge. Ask yourself this... How will you ensure that your best thoughts and actions will be remembered in 100 years?
It would be great to have sort of a FriendFeed-in-your-sidebar approach that you could access via your CMS. This would allow you to interact in real-time on your own blog while the data could be stored and archived in your own mySQL database. Yes, I'm picking apart the idea in my own head as I write, but the general idea is there.
The problems you mention are very true and due IMO to the "sliding window" nature of current real-time search. By that I mean the current search window is some fixed interval that keeps sliding forward and dropping off old entries. The end result is that real-time search is at present good only for things happening RIGHT NOW and even relatively recent stuff is quickly forgotten.
For mathematically inclined, one can say
Real_Time_Search(t) = dSearch(t)/dt
where dt is effectively the sliding window over which we are searching.
There is no real reason to do it this way, apart from expediency of initial implementations, perhaps. Real-time search should NOT be forgetting older entries. It should be definitely taking freshness as a MAJOR ranking signal but definitely keeping old stuff.
This way, assuming it is implemented well, real-time search should, with time, start converging to general search ala Google. That is actually the flip side of the above equation.
That's why we still have school books, right? Twitter and FriendFeed are the teachers during class.
Which is to say: Great article :)
Of course it all depends on your audience and who you want to reach, but I think this is definately food for thought.
Good post, and as always an interesting observation....