Scobleizer - Latest Comments in Should services charge “super users”?

Re: Should services charge “super users”?

Edvin Aghanian — Sat, 31 May 2008 07:53:23 -0000

Mr. Scoble,

I must say that I have not really cared enough to read your column in the past. After reading this post, however, I will make sure never to read anything else you choose to write. What I see is a person that is clearly ignorant about a complex set of topics related to application design and scalability, speaking sophomorically about them. Perhaps you should take some time, in your case, a great deal of it, and educate yourself about these matters before speaking great volumes of nonsense relating to the technical implementation of this or any other application.

Start by reading the comments here and questioning some of the very smart people, who have graciously taken the time to try and educate you. Please, for the sake of the thousands of people that clearly believe you to be an authority on matters of technology, stop this idiocy.

Re: Should services charge “super users”?

sam ism — Fri, 30 May 2008 17:31:47 -0000

Life is a very short journey and we are not here to waste it in using Twitter, Facebook, iPhone, Flickr,ETC.. then simple die.I just don't get it how do we benefits the community and the people by using these things what is the meaning of these things in life. people are dying all over the world and we are using Twitter to stay in touch with friends.

Re: Should services charge “super users”?

Steve Wilhelm — Wed, 28 May 2008 02:24:11 -0000

"First of all, Twitter doesn’t store my Tweets 25,000 times. It stores them once and then it remixes them."

I do know how Twitter is built. However, "back in the day," I was the development manager for a real-time, stock quote delivery system, so I do have some experience with architectural issues Twitter may be facing.

Let's look at the procedure Robert refers to as "remixes them." In the simplest architecture, there would be a single list (database, flat file, etc) of all the twitters created by everyone stored in chronological order. You may, as a storage optimization, just store a user id with the twitter string, and tweet time stamp (aka a tweet).

In this single architecture, a "remix" would require a query across all the tweets for a period of time for all people that a user follows. This query would be fairly fast when the number of tweets in the specified period of time is fairly small, and the number of users a person follows is fairly small. You can see that this type of query becomes more expensive when the number of users you follow increases and the overall number of tweets per period increases.

So to speed up this query, you could build some kind of index based on users. But maintaining this index would become expensive, especially during high incoming tweet periods.

So one might try to optimize this architecture by breaking up the universal store into list of tweets per person. Now each incoming tweet can be easily added to the user's tweet list.

Then the "remix" of tweets of the people you follow would require a join across each list and then sorted by chronological order. This would become increasingly more expensive when a user starts to increase the number of people they follow. It would be particularly expensive for super users who follow lots of users.

A reasonable compromise might be to keep a single universal stream of tweets in chronological order and two lists for each user: a list of pointers of all their tweets, and a list of pointers to all the tweets from the people they follow.

Maintaining these three lists would look something like: sender publishes a tweet, it is added to the universal store, a pointer is then added to the sender's tweet list, and then "push the tweet to followers" by walking sender's list of followers and add a pointer to the tweet to each "follow" list.

This approach scales fairly well. It allows the act of updating the follow lists to be partitioned across multiple servers. Each server can just take (using shared queues) a tweet from the universal store and "fan it out" to the appropriate followers. It also can be separates the operation from the inbound tweet processing.

To optimize the "fan it out" process, messaging publish and subscribe product like JMS or TIBCO Rendezvous and broadcast the tweets to the servers that manage follow lists. This would require a universal store process to publish all tweets and a cloud of follow list managers listening (aka subscribing) to tweet broadcasts updates for each followed person.

This approach also nicely addresses Twitter's need to separate outbound follower queues for users that have requested point to point delivery of messages via Instant Messaging and SMS.

For further scaling optimzation, you can have several tweet stores instead of one single universal store. You just need to ensure that all incoming tweets from a particular user are added to the same store to maintain ordered delivery to followers.

So it is quite reasonable to copy (at least references) each of Robert's tweets 25,000 times, just do so in a scalable manner.

Re: Should services charge “super users”?

BSally — Tue, 27 May 2008 16:35:17 -0000

Does it really cost that much to host someone's Tweets, even if they are quite frequent? If so, that seems like an inefficient model. I feel like most sites practice some form of economies of scale to where frequent users actually become more affordable (and therefore receive price breaks)... so, for example, Twitter might charge 2 cents for every 10 posts to relatively infrequent users like myself, and 2 cents for every 100 posts to frequent updaters. Keeping costs low for busy Twitterers would also increase business for them, since people are more likely to check your site frequently when it is frequently updated. If Twitter has difficulties understanding this, I can recommend a Basic Economics textbook (ebooks version).

Re: Should services charge “super users”?

Trevor Plantagenent — Tue, 27 May 2008 03:55:07 -0000

I don't see why not to charge the supertweeters, they pay whoever hosts their blogs, don't they? They're using Twitter as their personal blogging platform, no difference.

Re: Should services charge “super users”?

geekazine — Tue, 27 May 2008 03:54:49 -0000

Wow. Lots of interesting comments. Then again, lots of BLAH, BLAH, BLAH.

Here's a fact that people are overlooking. Traffic brings revenue. Let's say that the 25,000 posts get counted. What does that really amount to?

That means that 25,000 people are looking at what Robert Scoble is saying. If I was a person that wants to get out my product name, then I think I would pay Twitter to keep the service going. Better yet I might ask and pay Robert to push my wares.

About a month back I @(replied) Robert on something. I believe it was during one of many Twitters' "Problems". The exchange was short and sweet. However, I looked at my Followers an hour later, it jumped up 15 (which it doesn't normally do).

I tested the water by @ another person. The same thing happened. I gain more followers by replying to high profile twitters.

Now, apply that all to a marketing model. Communication can mean $$. I guess that's why Twitter was able to raise 20 million on it's own.

The problem isn't the Twitterflood. If that was the case then sites like MySpace and Facebook would be going down on a daily basis. If it DOES do what OM Malik suggests, then Twitter needs to look at their internal structure. Not at Robert Scoble, or Leo Laporte.

Limits and subscription fees are a great way to kill the idea. Some will pay for it, while others will say "See ya". Twitter will fall like a ball of flame into the Pacific Ocean.

They keep the idea fresh. To most, Twitter is an "Oh, I heard of that". People might know about it, but never signed up. Oncemore, Twitter can easily become a cash cow. The data that comes into Twitter is like when Daffy Duck found the Sultan's cave.

I'M RICH! I'M WEALTY BEYOND MY WILDEST DREAMS!!!...

Keep going Scoble. I'm listening...

Re: Should services charge “super users”?

pwb — Tue, 27 May 2008 01:37:15 -0000

Does anyone *actually* know how Twitter does what is does? Given the 140 byte limit, Twitter seems very "doable" with some basic design choices. This is why I think Ruby/Rails may really be a culprit here...it's too high level to support some of the things Twitter needs to do.

Re: Should services charge “super users”?

Jerry — Tue, 27 May 2008 01:27:39 -0000

Like they say, you get what you pay for. You want better SLA's and uptime? Crack open your wallet.

Re: Should services charge “super users”?

Alberto — Tue, 27 May 2008 01:03:54 -0000

With Nick's explanation on why de-normalization is needed to scale, it is clear one of the complex issues Twitter has to solve. The other one mentioned in this post is the business model, or when to charge.

Angus has the right idea - charging the followers - although I don't agree on the analysis. Still using Robert as the super-user, he should not be charged because of his tweets, but for the number of people he is following. Each tweet sent by the friends Robert is following will be copied on his queue (well the tweet ID) and the size and freshness of this queue (visually the 'With Others') can be used as the factor to charge.

Re: Should services charge “super users”?

Ethan Ambabo — Mon, 26 May 2008 23:14:06 -0000

Another aspect of the problem is that so many people use twitter in near-constant streams throughout multi-hour events so as to give "live" coverage of an event or just their stream of thought (case in point: the owner of this blog). I'm certain that taking all those SMS's in, displaying them, and broadcasting them out to webpages, RSS feeds, and then more phones is taxing to say the least. Look at the times it goes down and I'm sure you'll see a major correlation with tech events. I understand that some have found this useful, but I really don't and it's a sure fire way to get me to unfollow you, and I can guarantee why Twitter's user base hasn't diversified beyond "social techies."

Solution? Create a new service on top of twitter for twitter-streams, because obviously people don't get the idea behind 120-character limits (by the way, SMS has a 160-character limit) and hold their tweetstorms in a buffer to digest and spew out to followers when the server load can handle it.

Re: Should services charge “super users”?

Brian — Mon, 26 May 2008 21:32:46 -0000

"Sweet how you never had to work with an Exchange server which did exactly that, and then added ‘All’ as a recipient to the address book of every user.

Exchange never ever stored a message per user. If all users are on the same Exchange server and sent a message from someone on the same Exchange server, it it only stored ONCE. That's been the case since Exchange 4.0. Bedlam had more to do with people hitting Reply All to an alias that had users on different servers. It was the message queue that caused the primary problems during Bedlam.

In Exchange 2007 there is a deemphasis on SIS--it only applies to attachments. Not sure what the scaling problems are with Twitter as I have no idea how the system is designed. But, it would be safe to figure that whether or not they use SIS is not the source of their instability.

Now, back to your regularly scheduled debate about an non-scalable, useless communication tool.

Re: Should services charge “super users”?

Angus McDonald — Mon, 26 May 2008 18:55:14 -0000

Om's gotten it totally wrong. Twitter should charge your followers (past perhaps the first 100) for the privilege of getting access to your opinions faster. The website could remain free, but most twitterers I know use desktop clients to keep abreast, and so following someone is the only way to do that.

Charging 10 cents per month to each follower after the first would reap a far greater income for them, and annoy interesting twitterers less!

See my blog post for more:
http://falkayn.blogspot.com...

Re: Should services charge “super users”?

Prokofy — Mon, 26 May 2008 18:43:20 -0000

We used to have this thing in the '90s for people who wanted to broadcast to a large audience -- it was called "television". And we had this other thing you could use to interact with the broadcaster, called "the U.S. Postal Service". The fun part was when a guy like Dave Letterman would read your letter right there on the air, when he was broadcasting.

Re: Should services charge “super users”?

Tony Fendall — Mon, 26 May 2008 18:26:47 -0000

The whole debate is stupid! Why does no one seem to get it?

Could a team of competent software engineers build a system which could handle this many users? Yes!
Should twitter have a system which can handle this many users? Yes!

I don't understand why people are so keen to defend poor service. If it's broke, then the twitter guys should fix it. That means better code, more servers, what ever it takes.

If the problem is that they can't find a way to monetize it, then that's a different problem, but one where having lots of users should help, not hinder.

Re: Should services charge “super users”?

Tom — Mon, 26 May 2008 18:16:37 -0000

I just wanted to put in another "Everyone should read Nick Halstead’s comment" vote. Honestly, there are many things I admire about Scoble (getting the obligatory compliment before the insult out of the way) but this post is just ridiculous from at technical perspective.

As far as “charging super users” goes it isn’t really worth arguing because its going to be different for every service.

This is why you need a business model. To determine which ways of making money will be most effective and execute on them. Charging super users will be right in some cases while being wrong in others (depending on how much value the company in question can put into the “charged” scenario)

Re: Should services charge “super users”?

Nick Halstead — Mon, 26 May 2008 18:13:01 -0000

Could not resist typing up my complete thoughts on scaling twitter - http://blog.fav.or.it/2008/...

Re: Should services charge “super users”?

Richard Cunningham — Mon, 26 May 2008 17:32:20 -0000

Everyone should read Nick Halstead's comment.

Robert, I'm sure from reading these comments that the people talking about the technical problems understand how a the normalized databases they teach you in Computer Science course work. It just that large system can't use them (flickr for example doesn't it's sharded / de-normalized)

I don't think is twitter is sharded yet since they weren't at 350,000 users (http://highscalability.com/...
They certainly SHOULD be copying messages around if they are sharded.

You would think if they could get to 350,000 users on one database they could get to 1 million users by adding some database read-only slave servers.

Scaling isn't about saving disk space, CPU cycles, memory - that is being efficient it's not the same thing. Microsoft might try that with Exchange to reduce their customers hardware costs (not that it works from what I hear)

Scaling is knowing you can buy a rack of machines of servers and actually make them reduce your load.

Re: Should services charge “super users”?

James — Mon, 26 May 2008 17:29:17 -0000

First, as someone has already noted, any large e-mail system will indeed duplicate messages being sent to a large number of recipients — typically a single copy per server, with each recipient getting a Unix hard link (or equivalent) to that copy.

There are several critical differences between Twitter and e-mail, however. The push or notification aspect is one, but message size is a big one. In particular, each of the hard links pointing to a single instance of an e-mail will be bigger than the entire body of a Tweet! Duplicating messages, even in pathological cases like Scoble's, is trivial: 25,000 copies of a 140 byte message represents a mere 3.5 Mbytes, smaller than a single large e-mail body!

Similarly, I think you're overestimating the burden of keeping pre-calculated per-viewer data around: the default view has about 16 messages, each 140 bytes plus a bit of metadata (sender username/icon URL), total perhaps 3.2K. 10,000 users on the server? 32 Mb! Trivial. Even ten *million* users on a single node would fit on a PC you can buy online from Dell!

The best architecture is probably a hybrid: keep the recent message queue in RAM for active users (and update realtime when those they follow post messages), built the cache from disk when they log in. Even on a single host, with 15kRPM drives (4ms writes), that's 100 spindle-seconds; a pair of Apple's 16-drive arrays and you're looking at three seconds to process a Scoble-tweet, ignoring both write merging and RAID overhead.

In reality, of course, you can omit a lot of those write-barriers and re-issue the writes from a redo log in the event of a crash, cutting the write load still further. Mirror the writes and distribute reads consistently, you get failover and gain cache hits to boot (each server only sees half as many active users).

Or you write it all in Ruby and SQL then throw a kajillion dollars worth of hardware at making it all sort of work most of the time through brute force. Even $15m can only buy you so much brute force, though...

Re: Should services charge “super users”?

Larry — Mon, 26 May 2008 17:27:26 -0000

Maybe you're both right. Maybe it keeps a single copy of each tweet text and copies a tweet ID to each user's queue. The heavy-lifting of building the queue would be done at write time. To build a page it would look up each tweet ID in a user queue using a simple key-value map (which can easily be replicated and scaled.)

Re: Should services charge “super users”?

Robert Scoble — Mon, 26 May 2008 16:30:01 -0000

Trevor: well, if every Tweet is copied for every one of its potential readers, that totally explains why Twitter has some scale problems. Most Twitter users don't use the service very often, if at all (I watch).

In my scenario there ARE copies. Just not automatic ones. Also, Twitter only needs to keep the last 10 Tweets cached on each user's page, to keep the home page fast. Other pages take forever to load, so I doubt those are cached. Even in the home page scenario my Tweets would only be copied to those users who haven't had my Tweets replaced by other users (most of the time my Tweets would be pushed lower, so there wouldn't be 23,000 copies, only, maybe 1,000).

Either way, if I'm to blame for Twitter going down, why isn't FriendFeed going down? There's a lot more activity on FriendFeed surrounding my messages (and they aren't cached in any obvious way) and it's been down about 1/100th as much as Twitter.

Re: Should services charge “super users”?

JoeDuck — Mon, 26 May 2008 16:29:58 -0000

I think the charging issue is irrelevant because there are so few of you. If Twitter can't find a better way to monetize than smacking the <500 Superusers with huge monthly fees they are eventually going to be toast anyway. The Flickr model seems more realistic - charge heavy users a small annual fee and put them on a more robust platform. Heck, I'll pay just to keep having to hear everybody talk so much about Twitter, the challenges of which seem to have gripped the online community in a dangerously obsessive fashion.

Re: Should services charge “super users”?

Craig — Mon, 26 May 2008 16:19:40 -0000

I took a different approach than flat out charging users with mass followings:

http://thoughtindustry.blog...

Re: Should services charge “super users”?

Stuart Dallas — Mon, 26 May 2008 16:13:43 -0000

Scratch the 30 gig, misread Om's blog. Even so it would be a huge amount of data being moved about Twitter's internal network given that Scoble is just one of quite a few "super users" on there.

Re: Should services charge “super users”?

Stuart Dallas — Mon, 26 May 2008 16:10:13 -0000

I agree with all the commenters saying that Twitter must be copying messages to each follower, but I really hope they're just copying an ID 25k times and the actual text of the message maybe just a few 10's of times (i.e. to multiple caches). There's no way it would perform as well as it does (most of the time) if it were transferring 30 gig of data every time Scoble tweets.

Re: Should services charge “super users”?

Trevor Plantagenent — Mon, 26 May 2008 15:35:57 -0000

I don't believe this, you're making statements that are 100% false. Yes, every time you tweet, it's copied 25,000 times. It has to work that way or it wouldn't have scaled as far as it has. You're setting yourself up for massive humiliation when you're definitively proven wrong.