Yahoo's New BrowserPlus: Game Changer

Yahoo has introduced what I think is a revolutionary concept called BrowserPlus.

BrowserPlus is a browser plug-in unlike any other. As most of you know, a Browser plug-in allows you to add functionality to your browser that it is not built-in. For example, the most popular Browser plug-in is Adobe's Flash Player.

The reason BrowserPlus is significant is because it is really the plug-in to end all plug-ins. The idea is that once you have BrowserPlus installed, you can install different feature modules as needed seamlessly and without restarting your browser. Right now, installing plugins, while not exactly painful, is enough of a speedbump that, outside the Flash Player, getting people to install them is fairly hard.

The way BrowserPlus works is that if you go to a website that is leveraging BrowserPlus functionality, when a BrowserPlus based feature is needed and not yet installed, a dialog box opens up asking you if it is OK to install it. If you say yes, the dialog disappears and the new functionality is immediately accessible to the website. No restarting of the browser. No reloading of the web page. As Emeril would say... BAM!

The current feature set is light, but that is not really significant since what is really important about this is its dynamic structure. Some of the more interesting functionality available includes drag and drop to/from the desktop, a Ruby interpreter, and a text-to-speech engine. Note however that you can't work with it yet on your own websites because it is currently a "sneak peak" that is only being tested on Yahoo sites right now.

There are are few important points about this:

• BrowserPlus is like Google Gears, except with a functionally unlimited potential feature set.
• This dynamic plugin concept really could mean the end of waiting for the "next version" of a browser for many or most new features.
• This is a concept that could only be introduced by a small number of companies such as Yahoo, Google, or Adobe, because since they are offering browser extensions, it is critical that the source be trusted -- something a start up would have a difficult time achieving.
  
• If effective, BrowserPlus really could be a game changer for Yahoo. A reasonable deployment rate would put them at the center of the new web instead of just some popular website, which they are now. The strategic implications of this cannot be understated, and could fundamentally change the "who are we as a company" question that seems to be vexing Yahoo right now and forcing people to question their viability as a stand-alone entity.

In short, BrowserPlus is a big deal, and it makes me think Yahoo may just turn into a good long term bet.

Who Has Comment Copyright Ownership In A Disqus Era

There is obviously lots of discussion about copyright around things like music and video on the Internet, but another area that is going to be very important that is only recently getting attention is the ownership of comments.

This issue came to the fore recently when Robert Scoble commented on a post from Rob La Gesse's blog. The problem is that Scoble commented using Friendfeed instead of the standard blog comments. La Gesse and Scoble had a discussion where Scoble wanted him to move the discussion to Friendfeed. La Gesse did not want to do that, and at some point deleted his feeds from Friendfeed. This prevented the discussions about his blog from happening on Friendfeed. Unfortunately, as Mathew Ingram explains, this had the effect of deleting from public view Scoble's comments on LaGesse's blog. Scoble was upset that his comments had been deleted because he feels like he owns his comments.

The details of this story are laid out in more detail in Ingram's account of the matter, but the whole discussion got me thinking about the broader implications of who owns the copyright to your comments. Does the blog owner? Do you? What about when one comment will be viewed and under the control of more than one party, as in the case of Disqus. For example with Disqus you have the ability to edit your comments. And in some sense when you add a comment you are building a site for yourself (your collection of comments) and you are contributing to someone else's site.

In the past this was not much of an issue as blog sites have historically not given commenters much control after they hit the return key. And so the intuitive assumption though not necessarily the legally correct one, is that the blog owner owns the comments on their blog.

In fact though, copyright law gives broad protection to content creators, regardless of where they create that content. Certainly it makes sense intuitively that the blog owner would own the comment, but that may not be accurate legally. In order to perfect such ownership rights, it seems clear blogs really should have either terms of service that the commenter has agreed to, or a notice above the comment entry area that states who will own the comment being entered.

Since no blog platforms that I am aware of provide a mechanism for clarifying comment ownership rights, it seems to me that at best this issue is legally unclear, and at worst the site owner might only have the right to use and display the content in the very specific context in which the user placed the comment. So if, for example the blog owner wanted to use the comment on another site, or to reconfigure his blog in some substantial way, that might, theoretically require the permission of the commenter.

While it is unlikely a commenter is going to sue a blogger over something like this, at least in the short term, the lack of clarity is unnecessary and will at some point cause trouble for someone. Services like Disqus and IntenseDebate make the issue more complicated because they clearly provide control and display rights to two parties, the commenter and the blog owner where the comment is made.

This is a complicated problem with a simple solution. I would like to call for all comment systems to provide a mechanism to clearly indicate to users what rights they have and what rights they are giving out when they write a comment. Specifically, all that would be required is some clarifying text above the comment editing area or in a confirmation dialog box. This would be simple to do since a small number of blog/comment platforms handle most of the comments on the Internet including Google (Blogger), Wordpress, Typepad, Disqus, and IntenseDebate.

I would strongly suggest these major players address the issue, since right now we are in a comment copyright no mans land, and that can, over time, only lead to trouble.

On p2p: Adobe Says I Am Wrong. I Say... We'll See.

In a prior post I wrote that the new p2p functionality in Flash Player 10 would be a major threat to the CDN market. I still believe this is true, though I think it will play out differently than I imagined.

Adobe (finally) says I am wrong and there will be no p2p file sharing, and therefore no CDN killing, because Flash 10 will not be able to write files to the hard disk without direct user intervention.

Specifically this is what Justin Everett-Church, Sr. Product Manager for flash player had to say:

Will RTMFP enable massive file-sharing applications through Flash Player 10 beta?
No. Large-scale file-sharing network applications rely on the ability to read from and write to the hard drive constantly to only load parts of files into memory when needed. In Flash Player 10 beta you can load files from the desktop to the Flash Player runtime, but the file would need to be loaded manually by the end-user and remain in memory for the entire time the file would be needed. The drain on system resources would make this style of application impractical.

I had anticipated some user activity requirement being a potential tool Adobe might use for preventing good p2p, but not this one. If I had read the Flash 10 release more closely I would have caught this.

That said, I have two points.

First, as far as I can tell, this means that Adobe AIR applications should be fully able to create p2p apps in just the way I imagined, since they have full access to the file system. This in and of itself is a big deal because AIR applications are now deliverable through Adobe's one click AIR installer through Flash. It means that p2p applications are now trivial to write, and trivial to install. But it does mean pure Flash browser applications will have a harder time participating in such a network.

So the point is every stand-alone app that wants p2p will be able to have it. Of course doing an illegal Limewire will be harder because it will need to work like the original Napster, which had a central hub of control. Legally this killed Napster and would make it problematic for any would be pirate networks, which I think is as it should be.

So given that the only impediment for making a real CDN killing browser-based network is actually saving and reading files, the question is, why have such a limitation?

As I see it there are two potential reasons. I suspect one is the real reason and one is a fig leaf. The first potential reason is to protect CDNs who are major customers of Flash Media Server. This seems incredibly likely given how freaked out Adobe was about my initial article on this subject. The second potential reason, and I believe it is a fig leaf, is security. Clearly if you allow a browser app to write indiscriminately to the hard disk, you are opening the door to potential security breaches and this would indeed be bad.

As I see it, the answer to the security question is to allow every website that wishes, with permission of the user, to create a file system sandbox on the users hard drive. This area would not be accessible by other browser apps, and browser apps would not be able to write outside their own specific sandbox, potentially writing to other dangerous areas.

I am not a security expert, but it seems to me this sandboxing would prevent security breaches while allowing for lots of incredibly powerful data driven browser applications along with allowing real new forms of content distribution.

Why is this important? Because if we actually expect to build the Internet into a place that can do everything that television does and more, p2p must be a part of the equation. CDNs currently deliver one percent of the viewing audience of television in America. This is, of course tiny. I do not believe, using a CDN strategy, we will ever be able to deliver serious TV like experiences to the majority of the net. The math on that becomes problematic very quickly. But there is no reason we cannot move to a largely video based Internet once we have a secure, ubiquitous p2p architecture and move away from the centralized delivery model.

So I do hope that Adobe will tear down the artificial wall they have erected to keep browser based p2p at bay. Since everything is dependent of Flash Media Server they will still be able to monetize their work, but FMS will become much more broadly useful than it is today.

But even if Adobe doesn't do what I suggest, it just means smart entrepreneurs will make new AIR based environments or browser plugins that will allow people to write web apps that can deliver video via p2p.

As I see it the next layer here is tools that allow people to totally abstract the p2p functionality for use in content delivery. This is not too hard and I would imagine we will see at least one open source framework for the task not long after these tools all launch.

In any case it will be fascinating to see how this all plays out. The future is indeed ahead of us!

Arrington On Copyright: Wrong

I have always disagreed with Michael Arrington's position on copyright, which seems to be that people are going to steal, no one should do anything about it, and that copyright based businesses are going to die and that's not bad in his mind.

In January, Michael wrote:

Personally, I think a new era of free recorded music and paid live performances is a very good thing. Recorded music will become a marketing tool to get people to pay for concerts and merchandise. Overall the music industry will be smaller in terms of revenue. But the artists who are driven to create their art will continue to do so, and many will make a very good living from it.

First, to be clear if music goes down, so will every other form of copyrighted material including ultimately books, movies, TV, etc. What we will be saying is in the Internet era, copyright doesn't matter. And this is good?

Second, there is no evidence *at all* that free music on the Internet is an effective (i.e. successful career building) marketing tool. There have been no blockbuster successes that have come from, for example Garageband availability. I don't think you could even count more than a handful – if that – Internet-based artists making a living from music. I believe several of the American Idol contestants have been on indie music Internet sites, but you cannot attribute their success to the Internet.

Third, if the recorded music industry goes down, concert sales will not grow – they will shrink. This is because the money that goes into creating concert demand (all from record label marketing spends) will disappear. People *will* see fewer concerts and they will cost less money because of reduced demand. So not only will the recorded music business disappear, but so will the much of the live music business. So there will be no "live music windfall" to share. Revenue in live music will shrink substantially from where it is today.

In a continuation of the theme, Today Arrington writes:

My position is that it’s bad to criminalize natural behavior. And watching a clip of The Office, whether it’s legally on Hulu or illegally on YouTube is natural behavior. The only question is whether or not people are getting sued, or going to jail, for doing it.

There are several problems I have with the above statement. The first is that no one is criminalizing the viewer. In all p2p cases and in the YouTube case, the entity being criminalized or sued is not the viewer but the "facilitator". This means the person who posted or makes available the content, or the service itself.

In the case of Limewire, the software (I think by default) makes your music collection available to others. Most but certainly not all people know they are becoming *sources* of pirated content as well as consumers of it. But just to be clear, you can use Limewire without making your music available to others. In any case, I do not believe there has ever been a case of someone being charged with anything for receiving, watching or listening to pirated content. It is always for making it available to others in some way. So on the face if it, Michael's statement is misleading.

The second problem I have with this statement is that it reflects a misunderstanding of the purpose of law. I know Michael is a lawyer and I am not, but nevertheless, the purpose of law is most generally to stop people from harming each other, or society at-large, whether it is natural to do so or not. Certainly not paying taxes would be more "natural." And while we might argue over taxation levels, few would agree that allowing tax evasion would be socially beneficial.

The way it works is if the activity is harmful to others, we make the activity illegal. The larger the harm, or the more difficult it is to catch, the harsher we tend to make the offense in order to create a deterrent effect. This is the absolutely appropriate and necessary role of law in this country and actually more generally on our planet.

But the third problem I have with Michael's statement is that it suggests that he has some reasonable alternative in mind to monetize content published in this free-for-all way. But he does not. What he said today is this:

It’s time to rethink copyright laws, and it’s time for copyright holders to rethink their business models. The winners won’t be the companies that win or lose billion dollar lawsuits. It’ll be the companies that throw out everything that’s come before, and build new businesses around the natural behavior of people. Remove friction and win.

Without specifics, this is an empty, meaningless statement. Without enforcement and/or monitoring, whatever scheme one might come up with can't work. We don't currently have the technology to track every copyrighted piece of work, and we are unlikely to have any such technology in even the medium term. There is just too much content. Such tracking may be possible for TV shows and some music because of the severely constrained pool of content, but it will be impossible for everything else.

So generally, if we were to legalize the free trading of copyrighted works it would really be the end of monetizing content. Among other things, this would mean no more $100 million movies. Even $5 million movies would likely be impossible to recoup. Our visual entertainment would be confined to things like Star Wars Kid.

As I see it, the concept of "rethinking copyright" without specifics and without the willingness to follow things through to their natural conclusion is dangerous. This discussion must be about consequences. If you cannot propose solutions and provide reasonable answers to what the consequences are, such suggestions are only harmful because they embolden people to think that stealing intellectual property is acceptable and that IP protections are bad. But, In fact without intellectual property and attendant protections, we will be flushing down seven or eight percent of our economy directly, and indirectly twenty percent or more.

In other words, the stakes here could not be larger.

Interestingly, in the narrow confines of music, where usage could be tracked at least fairly well, Michael's concept of not punishing file sharing but turning it into a business might be possible. But this idea would, since there is no viable advertising model for music downloads, require that we tax ISPs and distribute the money in some way to copyright holders. But Michael is vehemently against that, calling it extortion.

The bottom line is you must do some combination of :

• stopping the illegal publishing of the content through laws and/or technology
  
• providing a means of monetizing free-for-all publishing through a tax of some sort
• finding an advertising model on the net that really works broadly – so far most advertising outside of search has failed (including YouTube) or is failing, and music download ads have failed horrifically.

The argument that copyright holders are wrong or stupid for not coming up with some new business model and for trying to enforce the law is not just wrongheaded, but as I said above, it is dangerous.

A big piece of our economy and our value in world markets is tied into the creation of intellectual property. The collapse of the concept of intellectual property will have devastating economic effects on everyone in every post-industrial economy. This may seem like it is just about illegal downloads, but the issue is much more serious and if not addressed portends an economic melt down of unthinkable proportions. A little "straight talk" is really critical at this point, because we really are talking here about economic Armageddon. And so I hope that real conversation can begin soon, before it really is too late.

Reasons CDN's Are In Trouble

The other day I posted about my view that p2p in Flash 10 will put a serious hurt on the CDN (content distribution networks) business. But that is by far not the only problem I see for CDN land.

As background, CDNs are services that provide high speed servers for hosting videos or other files that you expect to be accessed a lot, and that you want to be delivered very quickly.

The problem with CDNs is that they are inherently a commodity businesses. Yes, there will be niches for which they can and will continue to succeed. But for much of what CDNs offer, Amazon, and eventually Microsoft and Google (AMG) will do it cheaper.

Today, Amazon offers a service called S3 for Simple Storage Service, that is essentially a CDN, but without the high performance guarantees. S3, when its not down, has pretty good performance. People are *flocking* to it. I am totally confident that Amazon, with all its resources, will increase up time and get at or near CDN speed over time. They have invested heavily in all their developer platform services and I don't really see a way for anyone other that Microsoft and Google to compete with them doing what most will perceive as being a higher priced version of the exact same offering.

The reason AMG will have a cost advantage is obvious, but just to be clear, all three members have massive buying power that will be hard to compete with. And perhaps the most disturbing issue that must be facing the CDN world is that AMG does not need to make much, or perhaps any money at this business because they are all looking for developer relationships that will yield value in other ways.

Of course, if the CDNs really can offer unique services that AMG can't compete with, they will be less threatened. But I don't see how. The switching cost between CDNs is low, and there is no network effect or other good lock-in mechanism. So even though today there are performance benefits to CDN services from companies such as Akamai, Limelight, and Panther over lets say Amazon S3, I do not see those lasting. And I would even suspect Amazon and the other AMG members will ultimately offer different more focused products that aim more directly at the heart of the CDN market.

Finally, others are predicting a shakeout, because the entire CDN business had only $450 million in revenue last year while $281 million of venture capital poured into the market. In otherwords we are in a massive CDN bubble right now.

So I while I do see niche CDNs surviving for very specialized purposes, perhaps like live event Flash video streaming, I can't see CDNs really being able to make significant money at more basic stuff like hosting JPEG picture files or progressive download video (like YouTube) -- which are today the heart of the market.

As I see it, CDNs will have to be much more creative and really will need to move up the software stack to survive. This means offering tools that do more than just act like a big fast hard drive. They will need software tools that integrate with the storage that developers need. They will need to lock developers in by providing software tools that are unique and that get baked into the developer's application in ways that make switching hard, but also provide great value. This strategy is key because in most cases, the cost of development and time to market are far more important than the cost of storage and bandwidth.

This is my prescription, but I don't really believe the CDN guys have it in their DNA. We will see. But one thing is for sure. Whatever they are going to do they better hurry because AMG is coming and time is indeed of the essence.

Disqus And Profy -- First Step To Totally Portable Blogging

I am in Blogger jail right now.

And the capital "b" in Blogger was intentional. I mean the blogging platform from Google called "Blogger."

I feel like I am in jail because I want total blogging platform portability. I want a product that will allow me to suck in everything I have written in Blogger into an independent database. I want that for my comments as well. Then I want that database to be able to save my blog universe to Typepad, or Wordpress, or whatever.

Disqus, is an interesting commenting system that allows you to provide much more full featured threaded discussions in place of the default commenting system on standard blogs. Disqus is particularly interesting to me because their commenting system is totally disconnected from any specific blogging platform. As such, your comments are not in Wordpress jail, or Blogger jail or whatever. You can take your blog to another platform and your comments can come with you.

There are problems with Disqus such as the fact that if people search for a word that is in your comments those hits will go to the Disqus web site and not to yours. That of course is good for Disqus and bad for me. I have heard that they are going to fix that and I hope they do.

If they fixed that, to get me as a customer all they would need to do is be able to suck all of the old comments that people have made directly on my blog into Disqus. Then I would have a totally portable commenting world, and I would really love that.

All that would then be left for me to achieve blog portability is moving my actual articles. In this regard it appears that there are at least decent tools for moving from Blogger to Wordpress. I suspect other permutations are also possible, but this is, for me, only half a strategy.

One of the more interesting products in this regard is Profy. Profy is a blogging platform that has lots of cool tools that I havent yet explored. But the feature that really intrigued me was their concept of writing your blog in Profy and having it output to other blogging platforms.

When twittering with Cyndy Aleo-Carreira, Profy's blog spokesperson, I asked her about whether they were going to be able to go in the other direction and actually suck blogs into Profy. She did not promise but seemed to hint that might be something they are going to do.

If Profy did that, I think it would radically change the blogging landscape. It would make things much more competitive. I know that is the kind of thing that would really make me feel like I own my blog. Right now I feel like I write the blog but Google owns it. And so, the whole situation just leaves me with that "not so fresh feeling."

Adobe Refuses To Say I'm Wrong

In the above video, Kevin Towes, The Product Manager for Flash Media Server is asked by Andy Plesser at Beet.tv about my comments regarding the p2p capabilities of Flash and the impact on CDNs.

While Andy interprets Kevin's answer as being a refutation of what I said, it is clearly not. Kevin actually doesn't give a direct answer. He says "Hank makes some gross assumptions in all his diligence, its sort of interesting. But what we announced is something called RTMFP." But what he does not say is, "you will not be able to make a p2p network with Flash." Also, neither has the Flash Player Product Manager Justin Everett-Church been categorical about it here or here.

I have been wrong before. I will be wrong again. And it is certainly possible that Adobe has placed some artificial barriers that make it impossible to build a p2p network with their RTMFP protocol. I have discussed some of those here. But if I am wrong, it is just really strange to me that no one in the company will come out and say it.

A Detailed Five Step Twitter Scaling Plan

I don't want to suggest that the Twitter team doesn't know what they are doing, because I am sure they are, and have been, very smart folks.

And yet, the system keeps falling down (it seemed to be down most of yesterday), and there really is no good reason I can imagine for this to keep happening. I would offer myself up to help them solve the problem, but I suspect they probably have folks smarter than me working on it.

But I say all this to say I am sorry if I am insulting your intelligence, Twitter engineering team. It is not my intent. But I am going to suggest a simple structure that would help to solve your problem. If you have already thought of it or are implementing something better, Godspeed.

The essence of the concept is in the following diagram:



1. Automated CPU provisioning
First, in our work at Kloudshare, we use Amazon Web Services (AWS). You do not have to use AWS, but what you really *do* need is the ability to provision resources more or less on demand, or to just have enough extra server capacity lying around when you need it. The key is automated software that detects certain types of loads and brings on more computing capacity when needed. So you need, ideally, to be able to provision the CPUs on demand in software.

2. Denormalization
The critical concept behind most scaling is denormalization. If you are a database guy/gal you already know what that means, but if you are not I will just explain it in the context of Twitter. In my suggested architecture, every user's outbound tweets should be stored in a separate database from their inbound tweets. So when I look for all the stuff I have sent, it is in one place. When I look at all the things I have received, it is in another place.

In typical normalized databases you would want to store every message sent only once and not in two places as I suggest above. But while perfectly normalized databases make for "cleaner" databases, they cannot scale. Database purists often speak of the merits of normalization. And certainly some normalization is generally necessary. But in the new database world, a perfectly normalized database is an un-scalable database.

3. Sharding
In the above diagram you see that we have separated users into clusters. So we have not just stored "sent tweets" in a different database from "received tweets", we have broken each of these storage types into smaller groups of users. In our simple example, we store user data in clusters of 100 users. This is called sharding.

The purpose of this sharding is that when a user sends a tweet s/he will have no bottleneck in writing that data because only 100 other users are trying to write to that physical server. It can never be overloaded. That server then has the responsibility for writing the sent tweet to all the users that are subscribers to the given senders tweets as well as to the @ recipients (Twitter jargon for those of you not yet initiated).

The beauty of this is that each given database holding "received tweets" is not overburdened either. It only receives messages that are aimed at it. And it only is queried by the users who have their inbound tweets stored on that server.

What is great about this scheme is that it totally unburdens any given physical server and it is infinitely scalable. You could have a hundred or 10,000 such servers and performance scales 100% linearly with the addition of physical machines.

4. Shard Splitting
To be able to add CPUs smoothly, you have to be able to, do something I call shard splitting. In the example I gave above every database holds data for exactly 100 users. But in the real world each database should continue to grow until it starts to be over burdened. Then you "split the shard" into two databases. Each shard should know how to "split itself" when it becomes too busy. So if a database had gotten too slow holding the inbound tweets for 500 users, after the shard split there are two databases that each hold inbound tweets for 250 users each. This self splitting of shards allows for a very organic and automated type of scaling.

5. Distributed User Lookup
Finally, In order to know what machines have what users data there is a bit of overhead in storing a lookup table for each user that indicates where their inbound and outbound data servers are. This is necessary so that each server storing "sent tweets" knows what "received tweet" servers to send each tweet to. I suggest storing this table in memory, or on disk, across all needed servers.

This is a fairly light burden even if you are storing tables for tens of millions of users. I would suggest this table be synchronized across all the servers that need it using something like Terracotta, which is a specialized kind of "plug in" for the Java Virtual Machine that makes it so standard objects can be automatically kept in-memory and synced across multiple servers. In my estimation it is a very powerful tool in the scaling arsenal.

And so, that's it. Twitter scaling in five easy steps. Twitter in a box :)

OT: Race In Appalachia

David Gergen is a republican commentator that served in the Clinton White House. I have always had enormous respect for him as a straight shooting honest man. And nothing could be more honest than what he says in the above video about the Clintons. As background, in Kentucky, West Virginia and elsewhere in and near Appalachia, twenty percent of the voters in exit polls admitted that race was the reason they were voting against Barack Obama.

Gergen calls for Hilary to denounce the racist vote and to say that if that is the reason you are voting for me I don't want your vote. Wow.

I tend to be pretty hard on the younger generation. I think they have an extraordinary sense of entitlement. Among other things it manifests itself in cavalier attitudes around things like plagiarism, expecting high grades in school when undeserved, and indeed, music piracy.

But I must say, on the issue of race, I am extraordinarily happy to say that there is little racism in the younger generation. It appears that in thirty or forty years when all the racist bastards die off, we will really be able to start again. Unfortunately, we are not there yet, and Bill and Hillary Clinton seem perfectly happy to pour salt in the wound just to get elected. It appears all their historical work in the civil rights arena was nothing more than a political calculation around a powerful monolithic voting block.

What Gergen suggests would be a welcomed move towards repairing their horribly damaged image that they have created for the rear view mirror of history. But knowing what I now think I know about Bill and Hilary, such a gracious and proper move is almost inconceivable.

The Facebook Stat No One Is Considering

There are two things that are important in any Internet business: the number of users (NOU) and the revenue per user (RPU).

We are obsessed with NOU and NOU growth rates, and fundamentally uninterested in RPU. We do talk about CPMs as a kind of proxy for RPU but it is not. CPM is interesting, but it does not factor in how many actual ads it may be possible to sell. So if your CPM is $1000, but you can only sell one add a month and that advertiser only wants 1000 impressions that is only $1000 per month. If you had one million users that would not be an impressive RPU.

The other thing that is interesting about RPU is that it covers businesses that actually charge customers money as opposed to just businesses that sell ads. So I would suggest that RPU is the more interesting number, because it is likely easier to scale NOU than RPU. This is certainly historically true.

In any case, people always assume that lots of users means lots of revenue -- or lots of potential revenue. The most fascinating example of this is Facebook. Recently Henry Blodget has been suggesting that it might not be silly for Microsoft to buy Facebook for 15 - 20 billion dollars. Others have recently been suggesting that this is in fact Microsoft's plan.

I think the idea is ridiculous. Why?

Because as of today, we have no idea whether Facebook can get to any kind of a sustainable reasonable RPU. It is just assumed, and I think it is a *horrible* assumption.

The fact is in the last 10 years we have proven that it is relatively easy to get to a high NOU and *exceedingly* difficult to get to a really good RPU. Facebook has not proved that they have the magic to do it -- smart as Zuckerberg may be. I am not saying that they can't, but betting 20 billion dollars on the assumption that they can seems incredibly foolish. May I remind everybody that even for Microsoft, 20 billion is *a lot* of money!

The primary problem with Facebook is it is primarily a communications platform. And communications platforms are notorious for quickly scaling to high NOU while generating incredibly low RPUs. People don't want to pay for Internet communications, and they don't want ads in their communications streams. Even when they accept them, they tend to not pay attention and they have little ability to capture and leverage momentary intent in the way that search does.

I suspect strongly that the fundamental RPU of the social network model may be very very low, because the social network relationships that we are capturing are not economically valuable to us as users. This may be very different for LinkedIn -- particularly for recruiters and job seekers. But overall, I do not see anything that Facebook is doing that is likely to be really financially valuable to anybody. To assume they can just somehow get to a 20 billion dollar justifying RPU is unreasonable.

And so I suggest, whether its Facebook in specific, or Internet businesses in general, that we need to start looking seriously at the RPUs for businesses we talk about. High NOUs does not equal high RPUs. And one without the other just doesn't amount to much.

The Internet Causes A.D.D. (Attention Deficit Disorder)

No, this is not some new medical finding.

The point is this. With feed readers passing hundreds or thousands of blog posts by us a day, with Twitter and Facebook updates flowing like water, with email screaming for attention every few minutes, we have no more attention to give.

We can't pay attention to anything because we really give our full attention to very little any more. Apparently when we read a web page, we only look at around 28% of the words. I bet the same is true for reading print too.

This is why most advertising on the Internet is, on an impression basis, far less effective than advertising used to be. We know this mathematically because CPMs are so so so much lower than they used to be. It is also intuitively obvious if you are old enough to remember your interaction with advertising before the Internet. Advertising was very effective in penetrating our psyche and it no longer is.

Today it is almost impossible to get an ad message into my head. It is slightly easier, but just slightly easier in tech. But think about how hard it is for the typical consumer product ad to reach you today. Where could such a product be advertised and reach the average 18-40 year old person. They could try advertising on the Internet, but we know the click thrus are low and we know the clickers aren't predominantly in the 18-40 demographic. There is research, which I can't put my fingers on this second (the link is here - thanks Peter Christensen), which reflects that the people clicking on ads are older, and less Internet savvy. This should not be a shock.

The point is that we are parceling out our attention, not in bites, not in nibbles, and barely even in licks.

But the failure of non-search based internet advertising is really just the canary in the coal mine regarding our attention deficit. And while this is critically important to the future of our economic system, it is really, in the grand scheme of things, not nearly as important as what it suggests about our long term capacity for focus.

What does it mean that we have so little time and mental energy? How much time do we really think deeply about much of anything? Or perhaps we really are just using our brains more effectively today than we did twenty years ago? Could it be that we never really *needed* to focus so much and so now we are really just being more efficient and therefore effective with our time and our brain function?

That may be to some extent, but I am skeptical and a bit worried. I am not saying I have a well thought out reason to be worried, but I am. I just have this sinking feeling that we are losing something important when we pick up the pace of our activity, but reduce the depth of our focus.

Clearly the net has incredible benefits, but is this one of it's weaknesses? And if so what are the specific repercussions? I obviously don't have the answer here, but it seems to me that understanding the impact of this attention deficit is perhaps one of the most important philosophical questions of our time. What do you think?

Flash Product Mgr. Weighs In RE: Flash 10 And CDNs

The Adobe Sr. Product Manager for Flash responds to me regarding my comments about Flash, p2p and CDNs.

For those of you too lazy to read the whole thing, my 100 word translation:

We are not saying you can't do what you have said in your article. However, this was not our intent, and we don't like the idea that you are saying Flash is going to kill CDNs. We *love* our CDN partners and don't want any of them pissed off or scared. That said, if you do want to do any of this p2p stuff, you will have to authorize through the Flash Media Server. By the way, Flash 10 is *really* cool!

Full Text:

Hank,

The new RTMFP protocol in Flash Player 10 provides an alternative to RTMP allowing customers to develop real-time communication applications with an improved user experience. Applications like chat and games are great examples of what you are likely to see make use of this technology.

Adobe has made no announcements regarding Content Delivery Networks and file sharing. Adobe highly values its CDN partners they are an integral component in delivering rich video experiences to Flash Player. This new protocol has been designed for communication solutions, not massive content delivery.

We chose UDP because it’s an efficient protocol for low-latency streaming video conversations, voice conversations or other similar solutions. UDP combined with the high-fidelity of SPEEX audio, will allow our customers to build great voice and audio solutions targeting Flash Player 10. To enable RTMFP and the UDP transport, authorization from a future server-based technology such as Adobe Flash Media Server will be required.

Flash Media Server will continue to be used to stream massive amounts of content to Flash Player because it can serve and protect video and reach the largest possible audience. New information within Flash player 10 allows a developer to monitor how a stream is arriving and make decisions if a different bit rate will improve the video experience for the end user.

We are very excited about this new communication technology, the protection around it and being able to improve the delivery experience for both communication and media delivery to Flash Player.

Justin Everett-Church
Sr. Product Manager, Adobe Flash Player

Flash 10 p2p and CDNs – Deeper Analysis

Yesterdays piece on Flash 10 received lots of interest and generally a great response. However there were a few complaints about the fact that the Adobe press release didn’t actually say p2p, or that the Amicima technology was only about VOIP, etc.

One esteemed member of the Flash community and old friend, Brian Lesser, a Flash server guru and author of Programming Flash Communication Server suggested in the comments to yesterday's article that there might be restrictions on p2p usage that might limit some of the CDN possibilites I described yesterday.

On the other hand the piece did get some great coverage over at GigaOM from the likes of one of my blogging heroes, Om Malik, who essentially concurred with my assessment of the import of all of this. And so given the high level of interest, I thought it would be useful to do a deeper dive into the subject and respond to some of the commentary.

1. Adobe didn’t make any announcements about a CDN.

This is true, but it is not the point. My point is that developers will be able to build p2p networks on top of Flash 10. Flash 10 provides low-level tools that are the core of what p2p is about. NAT traversal and the general ability to connect one Flash client to another is all that is required. In this case, the p2p coordination of this will be based around Adobe’s Flash Media Server, or perhaps open source Red5 once they figure it out. In any case a server of some sort will be required for any of this to work to serve as the central hub for establishing connections.

We don’t know the details of how discovery will work, but presumably Flash clients will register with a server, and then it will be possible for registered clients to connect with each other. This core functionality will allow p2p CDNs to be built.

2. This technology is only UDP or only for VOIP and so can’t be used for files.

Bits are bits. I am quite sure that RTMFP (the protocol that is responsible for these p2p services) does not specify that it can only be used to deliver voice, and not data files or video. This would be stupid and based on the current Flash architecture, likely impossible. Moreover, as stated on their Wikipedia page, the Amicima technology before Adobe acquired it, was demonstrated using voice, video, files, and presence. Regarding the point that you cannot do files with UDP because it is lossy (i.e. designed to be able to drop packets), this purported limitation of UDP is misinformed.

Yes, UDP is not a guaranteed protocol like TCP/IP, but many p2p systems use UDP. The way such systems work is that the sender and recipient have to communicate about what is being sent and what is actually received. So if you are expecting 100 packets labeled 0-99, and packet 49 does not arrive, the sender needs to send it again. This is a *very basic* aspect of designing services on top of UDP. UDP is indeed a very effective low-level building block for anything, and is totally capable of supporting guaranteed delivery at a higher level of the design.

3. p2p will not be able to offer security for content on user's hard drives.

This is, just not so. Kontiki, introduced a highly secure p2p CDN in 2000. As far as I can tell there will be no impediments to implementing security since all that is required is file encryption on the hard disk (which is trivial) and transmission encryption, which is built into the protocol.

4. Couldn't Adobe's security model get in the way here?

It is true that we don’t know how Adobe is going to approach security. How restrictive are they going to be? In order for Flash 10's new RTMFP protocol to work for building any kind of CDN, it must be accessible programmatically (i.e. from Flash's programming language ActionScript) without user intervention.

For example, if Adobe requires that every time you want to establish a p2p connection between two Flash clients, you need to click on one of those Flash security dialogs then the whole thing comes crashing down. I could see if Adobe required that you click on such a dialog once for each website that required p2p access, but if they required that user interaction for every p2p access it would really be problematic. Aside from such draconian security measures, I cannot see any other impediments to implementing full-blown p2p CDNs. Of course any of you serious Flashers who have other thoughts please leave them in the comments.

5. Can this technology *really* kill CDNs?

Well, not in a literal overnight sense. There will still be a role for CDNs, but as I see it the heyday of growth, at least in their current form, is coming to an end. This is because companies like Amazon are commoditizing the central storage benefit through S3 (and will continue to do so) and because p2p technology will reduce, though certainly not eliminate the need for premium, centralized storage.

Of course the best such CDNs, using Kontiki as a design example, will of course also have servers that provide central storage and delivery for when the requested content is not available from another peer – a blended strategy. So the idea of centralize services will not go away. But killing growth is, in my estimation, the same thing as killing a category. If you do not grow, you die.

And so I believe, p2p is now poised to put a serious hurt on centralized premium server downloads. Of course it is entirely possible that CDNs will find some other significant value add, but as I see it this technology will take a huge bite out of the CDN market and for the CDNs that just ain’t good.

Adobe Introduces P2P Flash Player, Kills CDNs

Update: I have written a follow up to this article which is here.

Today Adobe announced the public beta of a new Flash Player that is going to change the way we all use the Internet. More importantly, the new player changes the economics of the Internet.

Interestingly the two really key features are not getting much play in the news yet as people have focused on the new graphics capabilities. But graphics is not what is really interesting here.

The first key Flash Player feature is a peer-to-peer (p2p) technology from a company they acquired called Amicima in 2006.

The second key feature is that Flash can now save files directly to your local hard disk.

These two new capabilities together make any type of p2p application trivial. Whether you are trying to roll your own LimeWire, or you have more honorable intentions such as implementing a server-less voice over IP (VOIP) application such as Skype, the new Flash Player will make these kinds of applications trivial.

What is even more interesting than cloning existing applications, is the innovation that will be unleashed by making p2p technology an assumed part of the web protocol stack. For example, it will be a few hundred lines of code to write an AIR application that will allow you to drop a file onto an icon and have that file appear on your buddy's computer.

But the most significant impact of all of this will be economic. As I see it, this is death to the content distribution networks (CDNs). The CDNs make their money providing companies the ability to centrally store and serve heavily accessed and/or large files. For years, companies like Kontiki and Red Swoosh tried to popularize p2p delivery networks that would make it possible to move large files without using such central servers. So if you wanted to watch a video or download a large game, these p2p systems would pull that file from other users who already had it instead of from the central store.

The problem with these p2p delivery networks was that not enough people wanted to download an application that only had the purpose of saving the service provider money. In other words, companies like Red Swoosh and Kontiki didn't have enough end-user benefit to get people to download them.

But because people download Flash anyway, the chicken/egg problem associated with getting the p2p software onto people's computers is eliminated.

What all of this means is every major video site will now be able to deliver video streams, while using only a trickle of bandwidth. When the Skpye founders introduced their video service Joost, this was what they hoped to be their secret weapon. They wanted to operate a video network in the same way they operated Skype, i.e. at very low cost. But people preferred to watch videos in their browser without downloading anything special.

But baking p2p into Flash means *all* web video sites will be able to deliver video using minimal central server bandwidth. As a result, the largest and best CDN customers will disappear, or at least will radically reduce their needs because, in effect, every video website will have the low cost of operation of Joost.

Of course this is not just about video. If you have any large file movement needs that require a CDN, Flash-based p2p will be on your short list of engineering imperatives.

This new version of Flash, combined with Amazon's super low cost S3 storage system will be fatal to many CDNs. Personally, I'd be shorting every public CDN stock around the time that Adobe leaves beta with this stuff, because it spells real trouble for that market.

And if you ignore me now, don't say I didn't warn ya. For the CDN market, this is Armageddon.

Microsoft Shepherding Intellectual Property Discussion

As someone who has been around the computer industry long enough to say that I was there before the birthing of the microcomputer (the term we used before PC), my view of Microsoft has always been dim. The reputation of Microsoft as a company acting on the edges of propriety is legendary. There are many examples, from accusations from GO Corporation's Jerry Kaplan that Microsoft stole the core concepts they pioneered in pen computing in the early 90’s, to Microsoft’s anti-competitive arm twisting of PC manufacturers.

When Google was formed, they took as their motto, “Don’t be evil.” This was clearly code for “we won’t be like Microsoft.” And certainly, at the time, history was on their side.

But indeed times have changed, and some might even suggest the roles have reversed.

Windows is clearly not gaining ground in devices or phones. Web applications using HTML and Javascript, and Adobe’s Flash and AIR have almost totally supplanted Windows as an application development target. Microsoft’s web strategy has, for the most part, despite spending billions of dollars, been a failure.

To quote Bill Gates, “Success is a lousy teacher. It seduces smart people into thinking they can't lose.” Indeed, if Gates is right, in recent years Microsoft has finally had the opportunity to do some solid learning at the hands of Google.

The tipping point for the change might have been the loss of the anti-trust case. Or perhaps it was tied to the retirement of long time Microsoft general counsel Bill Neukom, and the ascendance of kindler gentler general counsel Brad Smith in late 2001. But whatever the cause, the result is clear. Microsoft, right before our eyes has become a very different company from the one we knew ten years ago.

The first indications became clear when, on the arrival of Smith, Microsoft began settling cases that they never before would have settled. The old Microsoft fought everything tooth and nail to the bitter end. Microsoft was famous for not giving an inch – on anything. Ever.

More recently, Microsoft has indicated a desire to turn another corner. Tom Rubin, Chief Counsel for Intellectual Property Strategy at Microsoft, gave a speech last week in New York about the future of intellectual property and creativity online. The core of his argument, is the basic idea that technology companies and content companies need to figure out a way to get along. He frames the issue as primarily a social and structural one more than a technological one – a point with which I wholeheartedly agree. The term he uses for this needed awakening is “stewardship.”

Tom’s basic point is that recent data is showing us that the Internet is currently not a platform that can economically sustain content creation. As such we need to figure *something* out to allow creative efforts to remain economically viable. Tom defines stewardship as follows:

“So stewardship means appreciating the impact of one’s actions on others and on the media ecosystem as a whole. It means making sure that new species of individual creators can thrive and survive, and that new technology is given room to flourish, even if they both challenge your traditional ways of doing business.”

After the talk I had an opportunity to sit down with Tom, and I discovered that the speech reflects more that just a philosophy about intellectual property. The speech is really a marker for a new concept for how to approach intellectual property by Microsoft. It reflects a desire to truly move the discussion forward in some substantive way. Microsoft wants, in some respects, to be a shepherd. As Tom explains it, Microsoft’s desire is not to impose what is right, but to engage us all in a discussion that leads to a future we can all live with.

Tom pointed out in our conversation that Microsoft is both a technology company and a content owner, and so they do not take the perhaps extreme positions that pure tech, and pure content companies tend to get into. Of course, the fact that they are Microsoft, will indeed, in the minds of some, taint their credibility.

But what is really fascinating here is the idea of Microsoft *seeing themselves* as a kind of reasonable, thoughtful, verging on academic voice on this issue – almost an honest broker. (Yes I do see the irony in that last sentence.) Of course it is not at all clear that this will be possible. It is entirely possible that such talk will not actually fly internally at some point because of competing interests. It is also possible that the outside forces are so entrenched and Microsoft’s history hopelessly undermines some kind of new role in the industry. But the fact that this kind of talk is even happening is hugely significant since it would not have happened in another era.

At the end of the day only time will tell if Microsoft really can bring everyone together to come to our collective senses. But it seems to me some kind of intellectual property rapprochement is critical to our economy, as well as to our culture. And so, in my book, Herculean though the task may seem, it is certainly an admirable and worthy effort.

And so, perhaps for the first time in my professional life, I can actually say, at least on this issue, I truly applaud Microsoft. Godspeed.

Blaine Cook Writes First Blog Post Since Twitter... About Scaling

Today, Blaine Cook, formerly Twitter’s chief architect, writes his first blog post since leaving Twitter. It certainly seems like he has some stuff to get off his chest.

The gist of the piece is that languages don’t scale, architectures do. It seems clear why Blaine might want to say something about scaling, since it is clearly the number one issue at Twitter. And he did take some heat for the problems there. And in some corners of the Web, Ruby, the language that Twitter runs on, has taken even more heat.

Blaine's arugment: Scaling is fundamentally about the ability of a system to easily support many servers. So something is scalable if you can easily start with one server and go easily to 100, 1000, or 10,000 servers and get performance improvement commensurate with the increase in resources.

When people talk about languages scaling, this is silly, because it is really the architecture that determines the scalability. One language may be slower than another, but this will not affect the ability of the system to add more servers.

Typically one language could be two or three, or even ten times slower. But all this would mean in a highly scalable system is that you would need two or three or ten times the number of servers to handle a given load. Servers aren't free (just ask Facebook), but a well-capitalized company can certainly afford them.

The problem comes when your architecture is such that you can’t just add more servers. While Blaine does not discuss this, the primary reason things don’t scale has to do more with the cost of data access. Databases are almost always your bottleneck, because all your data typically needs to be stored in some central repository.

So how you architect your data storage and access will determine your scalability. For example, do you use RAM based caching like memcached to improve performance and limit the need to read the database? If so, is your caching architecture good enough to limit most reads from the database, or just a few? These are the kinds of architectural decisions that will determine system performance.

In Twitter's case, there is zero chance that the problems there are in any way related to their language. It is likely that there are architectural challenges which come from the fact that it is very hard to cache a Twitter data request since no two people ever get the same data. And even for a given user, the data requests change quickly since users are always receiving tweets. This is a hard, though not unsolvable problem that requires a very specialized caching architecture. Eran Hammer-Lahav, has done some interesting work in this area and talks about it in an extensive blog post.

The bottom line is languages don’t kill scaling, programmers do. As such, Blaine's piece, while sounding a bit defiant, might really be read more like a mea culpa. Though, to be fair, despite all the chatter and criticism, scaling Twitter is indeed a non-trivial problem.

Media Darling Powerset vs. Non-Media Darling Hakia

Over the weekend, The web was abuzz with discussion about Microsoft considering the acquisition of natural language search company Powerset. Today there is yet more coverage. Some time ago I had heard a rumor that someone was looking at Powerset, but was relatively uninterested. Hearing that the potential acquirer is Microsoft certainly makes it more interesting, but I have to say the concept leaves me more than a bit incredulous.

From skeptic to user

I became familiar with Powerset's only competitor, Hakia initially because they are a New York company. I became intrigued with Hakia because several months ago I tried their search engine, and it worked – really well. This was a surprising result for me since I have always been a skeptic regarding all things relating to artificial intelligence, speech recognition, natural language processing, and other such fuzzy technologies.

At least in the area of natural language processing Hakia that has changed my mind. In fact, it has become common for me to use the Hakia search engine when Google does not deliver sufficient results.

Hakia and Powerset are part of the same general area of natural language search. The idea with both services is that you can actually ask specific questions and get answers. But there are critical differences between Hakia and Powerset. And those differences bring me back to my incredulity at the idea that Microsoft is taking a serious look at Powerset.

Powerset indexes 750 times slower than Hakia!

I have no expertise in natural language processing or semantic search, or any type of full text search for that matter. But as far as I can tell, Hakia’s technology is *far* superior to that of Powerset’s. Why would I say that?

Well first, as I have already said, it works. It is a real live search engine. I use it. I can’t say the same for Powerset. Powerset has yet to show anything but a search engine for Wikipedia. A big part of the reason Powerset doesn’t seem able to offer a real search engine is the fact that according to their own reports, it takes them about 25 seconds to index a page, based on an average of 25 sentences per page. According to Hakia it takes them 1/30th of a second to index a page. Essentially this means that Powerset cannot scale. It is seven hundred fifty times slower than Hakia!

Now you might assume that Powerset is slower because it’s applying some serious, and superior indexing mojo, and therefore what it is doing is much more valuable than what Hakia is doing. But alas that is also not true.

Hakia really knows how to read

Hakia is doing something called “ontological semantics”. What this means is that over the last four years, Hakia has developed an “ontology” for human expression. In layman terms, what this means is that what Hakia does when it indexes a page is to look at each sentence and figure out what the *questions* are that each sentence answers. Any given sentence usually answers 3 or 4 questions. These questions are coded and go into what Hakia calls their Qdex, or question index.

In order to be able to figure out what the relevant questions are for a given sentence, Hakia’s indexer has to literally read the sentence. By “read” I mean it has to understand the actual meaning of the sentence semantically. This is a big deal.

Powerset uses statistics + syntax but can’t actually read

So, while Hakia is actually reading, Powerset, does not actually attempt to understand what sentences mean. It uses a system that parses the syntax of the sentence and guesses matches based on statistics. But this approach means that for questions that do not match previously encountered syntactical patterns, the system will not be able to find answers, even if there are in fact answers in the database.

Powerset benefits from the Silicon Valley echo chamber

Now, if, for a moment, you presume that it is true, or even *possibly* true that Hakia is the superior service and technology, or if you even assume that Hakia is just equivalent to Powerset, why would Powerset be so continuously celebrated while Hakia is overshadowed?

The only answer I can come up with is that the west coast is such an echo chamber that very little sound gets in or out. And so it must be shocking when a New York company develops a technology that seems to beat the pants off something that should be pure Silicon Valley. Just a thought.

In any case, it seems, for the record, worth noting that we have the clear leader in natural language processing and search technology right here. And, as an admitted New York partisan, after a while it does get a little annoying to hear such continued fawning over a west coast company that is very likely, at the end of the day, just another Silicon Valley also-ran.

The Stats Explain Why We Don't Read Banner Ads (And What We Can Do)

Jakob Nielsen recently published two interesting research statistics that help explain why display ads don't work.

First, Jakob says we read a very small amount of the text on most pages. Apparently we only, on average, read at best 28% of the words on the page and most likely around 20% of the words.

Second, in an older article from late last year, Jakob also says users avoid anything that looks like a banner. This may seem obvious, but the interesting thing to consider is that all web pages on a given site (and even across the web on different sites) are likely to have the same pattern for what looks like an ad vs what looks like content. Your eyes quickly learn that anything that is on the sides, or anything with a border around it, or anything that looks photographic is an ad.

Newspapers and magazines generally do not have any such easily to discern pattern. Pages are generally laid out manually by a designer on an issue by issue basis. Smaller ads are blended in with the content, breaking columns and such so that advertising will have unavoidable and unpredictable proximity to content. Moreover, many ads take up the full page which requires at least some minimal scanning before turning the page. And because the ads are often so big, you almost can't avoid at least seeing a bit of it before turning the page.

The bottom line is that users see too much stuff on the web and so they focus on very little. On top of that they have learned effective patterns for avoiding ads. And while we try to avoid ads in print, the physical size of the advertising, the unpredictable patterns, and the high degree of mixing advertising and content make pattern based ad avoidance much harder in print.

In my view, the model for how we organize information on the web is fundamentally broken as it relates to siphoning off attention for advertisers. We need to inject more randomness into our designs. Indeed, the kinds of thing that Jakob Nielsen promotes as improving usability do in fact promote usability while *reducing* monetizability.

Magazines and newspapers are *not at all useable*. I am often frustrated that it is very difficult to find a table of contents in a magazine, or that some pages are missing page numbers that would help me find an article. This is all on purpose!

Print publications are very effective advertising platforms in large part specifically because they place "attention speedbumps" in your way. If print publications were as streamlined as websites, with ads neatly presented off to the side, ads wouldn't work there either.

And so we have a decade of smart folks like Jakob teaching us exactly the right stuff to make our sites easy, but exactly the wrong strategies for making money. Amusingly, at least a part of the message is that a little disorganization is absolutely critical.

I Am So Sick Of "Social"

I have come to *loathe* the word social.

It has lost all meaning. Everything is friggin' social. Social meeedia they say.

Why not just be done with it and call email "social messaging" and instant messaging "instant socializing". How about "social walking" for when you walk down the street and say hello to people. "Social eating", for when you take a friend out to eat and are nice to the waiter. "Social sleeping", for when you go to bed with your wife or significant other, or, well with whoever. How about "social peeing", for when you (guys) go into a public bathroom and there is someone at the next urinal.

We use the term as though we are just so amazed that we can use the computer to actually *communicate* with people and/or share experiences. Well knock me over with a darned feather.

Now what *would* be amazing is actually figuring out a way to make some money out of all this "socializing". That would be a neat trick, since it seems even Facebook and Myspace are having a hard time with that -- at least on a revenue per user basis. And what is fascinating about it that we keep being told that it *will* make money. We are going to figure it out, etc.

I'm calling b.s. on that.

I don't think we are going to figure out a way to make significant dollars off of social *anything*. Yes the definition of significant is up for grabs. But for me it just means a major segment of the computing space. Like games. Or search. Or computers. Or enterprise software. Or like anything more than a financial footnote.

Indeed all this social may be a great public service. I personally love meeting and talking to my readers. And it may be fun to tweet, and message, and comment and such, but I don't think there are more that a few pennies per user per month in it. And for some services which get really really big, that will be fine. But it ain't no friggin' revolution -- financially anyway.

But anyway, enough about the money. That's a tangent (ironic isn't it). Can we please just stop calling everything social? Pretty please?

Thank you.

My Fascination With Reducing Development Iteration Cycles

I have, of late, been thinking a lot about the issue of how to make money on the Internet. And I have come to some very specific thinking about how I am wired, what appeals to me, and how that intersects with Internet money.

I like making tools that help people make things. I have primarily, in the past, made consumer technologies, but I think the idea of figuring out how to abstract my own processes has always been of interest to me. I think that one of the most valuable things you can do, whether its for yourself or for others, is figure out how to reduce the amount of time to go from an idea to an execution. Particularly in todays world, velocity of iteration is critical.

And so, if you can create tools for yourself that allow you to iterate quickly in whatever business you are in, that is incredibly powerful. If you can respond to new customer needs and requirements twice as fast as your competitor, that is an awesome advantage. It has always been of interest to me to try to build my products in such a way that this type of fast modification and enhancement would be possible -- not that I have always succeeded. But before my current efforts, I have never actually focused on creating tools for others that would increase their ability to iterate faster in *their* creative process.

Today, this idea seems more relevant than ever. When we are experimenting with business models, the one thing we need is to let the technology get out of the way so that we can figure out what is actually useful. Being able to blur the line between building a prototype and building real production software or services is critical to experimenting with and figuring out the much more consequential business model issues.

Indeed the world is clearly moving in this direction. But there is so much further to go. Figuring out how much smaller we can make iteration cycles for services that are not just prototypes but real scalable applications is what I find really fascinating right now. It seems to me it just might be profitable too.

Come on Already, Isn't *Someone* Going To Buy Adobe!

Robert Cringley has been recently speculating that Apple will buy Adobe. Jon Gruber doesn't think its going to happen. On the other hand, Blogger Matt Maroon has written that Microsoft should buy Adobe.

My view is that *someone* should buy Adobe.

From a Microsoft perspective, even with a premium, this is a deal that is smaller or perhaps at worst case similar in size to the Yahoo deal. But it is much more strategic. Microsoft's Silverlight efforts will likely never gain the kind of traction that Adobe has with Flash. It would not be at all hard for Microsoft to take their WPF platform and port it to the flash engine. This would allow multiple development targets for the same runtime, meaning developers could write in Actionscript and Flex, or they could write in C# and use a Microsoft's WPF interface framework. I believe this is possible because several years ago, the company now called Aptana, before deciding to switch strategies and names, had a similar product on the market called Xamlon. There is another product currently in the market called GOA Winforms that does the C# part and implements the standard .NET GUI interface framework.

The point is that the Flash engine is a work of art not easily bested on any front by Silverlight without years and years of investment. In that time frame, Flash will be getting stronger and more broadly deployed through things like the Open Screen Project.

From Apple's perspective, Adobe would indeed be a big acquisiton based on their history. Apple's market cap is $162 Billion and Adobe's is $21.4 Billion. But depending on the premium required to get a deal done it would likely still be accretive to earnings. But the real opportunity is strategic. For Apple, this would be a way to inject "Appleness" across the entire Internet.

Apple is currently a niche player, albeit a highly profitable niche player, in the desktop and the phone worlds. Buying Adobe would give Apple a chance to be *the* brand on the Internet. I believe that Flash and AIR are destined to be, effectively, the new operating system of the Internet. Flash is and will continue to be the leading cross-platform API on the Internet, but its developer mindshare, big as it is, is growing.

For Apple to own the platform that could be the gateway to writing apps that run anywhere, but just run better on the Mac would be an incredbily powerful lever in taking Mac OSX (on the iPhone and the Desktop) to the mainstream in an incredibly fast manner.

But the main reason I would be trying to buy Adobe is that if Microsoft were to buy it, it could be disatstrous for Apple. Microsoft has done a horrible job in keeping its developer APIs relevant to mainstream developers with the advent of the Internet. An acquisition of this type could bring Microsoft the kind of dominant API position that they had in the 90's.

Of course, this doesn't at all fit the style of Steve Jobs. He has never done a major acquisition, and has a major case of NIH (Not Invented Here) Syndrome. Somehow I just don't think Steve's ego could handle something like this. And with Microsoft having just ditched on Yahoo, I suspect it will be a while before they make any other big bets like this.

All that said, it is still amazing that Adobe is being allowed to take such a position in the marketplace. Adobe has a broadly distributed platform at around 99% penetration, an incredibly well tested well engineered technology, great engineering management, and a huge developer base.

In any case, in my view, investors that are long Adobe should be happy. Adobe is well poised to join the really big boys whether they get an official invitation or not.

Killing Twitter Before It Can hurt Us

This weekend there has been a fair amount of talk about the idea that Twitter is too important to be relied on as a centralized, non fault-tolerant platform. Both Dave Winer and Michael Arrington at TechCrunch are talking about the problem and how to fix it. The argument is that Twitter is not distributed. This means that if Twitter goes down no one can Twitter. It also means if Twitter goes out of business, or just screws up, we could lose all of our tweet history.

The idea being posited is to make a parallel open source run "twittersphere" that supports the Twitter API -- lets call it Twitshadow. Twitshadow is built to be a shadow of Twitter that all third party apps can use. The shadow could even be *the* way to talk to Twitter by implementing what would essentially be something like a write-through cache. Third party apps write to Twitshadow, and Twitshadow writes to Twitter. Or it could just be a second call that all third party apps make along with writting directly to Twitter. When Twitter goes down, this open source non-centralized Twitshadow keeps on chugging. And the more time that passes, the less important Twitter actually becomes because most of your data is in Twitshadow. In fact Twitshadow could even support sucking in tweets from Twitter so it could be a full replacement of Twitter, history and all.

This whole conversation is fascinating to me.

It is entirely possible that before Twitter makes its first penny, it will become too important to exist in its current form, and the community will feel it has to be replaced by an open source distributed framework. This should strike fear into the hearts of anyone who decides open their API. While the Open API strategy has clearly worked in terms of adoption, it may have worked too well. In fact it may have worked so well that Twitter may be killed before it has even really made it out of the womb, by people that find it so important that they can't afford to really have it be a company.

Technically, Twitter is ripe for destruction by open sourcers. It is a super simple API. It is loved by lots of really smart people that definitely have the intellect, the means, and the motivation to create Twitshadow. In fact open source will tend to be far more effective a development model for something like Twitter than closed source, in large part because a proper solution to this is a very pure computer science problem. This means the open source community will almost certainly be more clever about the solution that one tiny closed source company because there will be lots of heads focused on an easy to understand problem.

The lesson from all of this may be that communications apps can't live without an open API, But they can't live with them either. Of course I have been skeptical that any communication app can make money, and particularly Twitter, but I could not envision that they would and could be undermined as a platform like this. It is truly astonishing to watch.

We Just Don't Know Everything

One of the things I am fascinated by is the desire in human nature to explain everything with a pattern and to dismiss things that don't fit in a box. What got me thinking about this was a discussion on Yahoo about why American Idol's ratings are down.

There were all sorts of rationales posited, but the thing that is fascinating to me is the belief that there must be some systemic rationale for the drop. Couldn't it be that the contestants were less interesting this year? Or could it be that we are just a bit bored with the brand and concept? I don't know, and that is exactly the point. Too often we demand a grand explanation for things that may be unique to a specific situation and do not fit a pattern.

Lets take a more controversial subject, at least in tech crowds. God.

What is fascinating to me is that smart people can be so confident that there is no spiritual plane of some sort. Some level of existence that we just don't understand. My purpose here is not to convince you to believe in God or to believe anything about American Idol, but to suggest exactly the opposite. I think that we should all embrace more than a little bit of uncertainty in life. Trying to explain everything with our severely limited perceptual abilities is foolish. I am not saying that we should stop trying to figure out the answers. But on the other hand acceptance of the unknown and the, at least currently, unknowable is incredibly freeing. I feel like we as technologists often find this idea discomfiting. I personally find the quest for knowledge combined with the certainty of my uncertainty to be a very satisfying balance.

Adobe Takes Gloves Off In Mobile World

Today Adobe announced a new hugely important strategic initiative regarding the Flash Player called the Open Screen Project. It relates primarily to the mobile world but also to the embedded device world in general.

There are three keys to the announcement. First Adobe is removing all licensing restrictions and fees for the Flash Player and the SWF file format that the player runs. OEMs will now be able to embed Flash at no additional cost and with no restrictions.

Second, the next version of the Flash Player and the AIR runtime will have two separate parts. The layer that is platform specific will be separate from the layer that is the real brains of the Flash system. This means that it will be easy to embed the Flash player any new non-PC device offering. All you will have to do is port the platform specific layer to your device. And Adobe will be providing a reference implementation so that you can see exactly how to do it, presumably just making minor changes related to the specifics of your platform.

And finally, the next version of the Flash Player will be the same across all devices. This means that you will be able to use Flex, Flexbuilder, and Thermo to build applications that run on mobile and embedded devices. This will *radically* simplify the process of making applications that run on mobile devices. It really is, essentially *the* next generation mobile operating system. No its not an OS in the traditional sense. That will in general continue to be some form of Unix or more generally Linux. But the Flash platform will become the premier application development target for devices.

This is a direct shot across the bow of both Apple with the iPhone and Google with Android. Adobe has far more 3rd party developers than Apple does with Mac OS/iPhone or Google does with Android, and if they can make it totally seamless to develop for desktop or mobile, it will radically change the dynamics of the business. Presumably Adobe will be able to port this next version of Flash to the iPhone as well, though the politics of that will be interesting given Steve Jobs' antipathy for Flash.

From a business perspective, Adobe seems to have everyone onboard that matters including Nokia, Motorola, Sony Ericsson, Qualcomm, Samsung, Intel, and lots of others. Now that Flash is free and presumably easy to embed, it instantly becomes the mobile and embedded software platform to beat.

Seven Do’s And Three Don’ts For Creating New Web Products

Yesterday I wrote about how everyone else seems to be talking about how lame the new product ideas that people are developing are. I had at least one commenter over on Hacker News who said, in effect, that it was useless to comment on such stuff because being creative is *hard*.

My first reaction was, duh!

But my second reaction was that perhaps I could provide a better service by helping people actually think about the process of idea generation. First, I wrote a rather detailed article in January called how to create great ideas. In conjunction with this article, if you are looking to make a new truly creative product concept I suggest you read it.

With regard to my do’s and don’t please be aware they are guidelines. There are no hard and fast rules because there are always exceptions. But the rules, combined with the logic behind them should be helpful in thinking about how to actually create something useful. As discussed in my how to create ideas piece, the key element here is iteration. Your first idea may be interesting, but it’s rarely in and of itself good. Good idea development requires work and iteration usually with feedback from others in the loop somewhere.

Also note that some of the rules are mutually exclusive so clearly use common sense when interpreting the list.

The Don’ts

1. Don’t build a communications application.
Communications applications are deceptive. They have an incredible ability to get big fast because they are, done right, inherently viral. Having large numbers of users is always appealing to investors because there is often some fat cat corporation that feels they can monetize your communications app successfully. The tide on this I think is changing.

The problem is that communications apps never make money. You can’t charge people because then it won’t go viral. You can’t run ads because people don’t want ads in their communications application. Of course if you can do something easily (like twitter) and go viral I guess I would go for it, but if you do, you should be prepared to head for the escape hatch as soon as possible. Just be aware that as far as I can tell, for the most part these are not real businesses.

2. Don’t build a social application.
Most of the rules regarding communications applications apply here. Social apps can go viral, but they have all the same problems as communications apps except that they *can* be monetized if you get very, very big. The CPMs are very low and so running ads is very difficult until you get to a massive scale. Moreover, today the market is so saturated it is hard to imagine a social application succeeding if it does not hit one of the below “do’s” out of the park. Indeed of you can marry something social to something where there is a real path to revenue then you have indeed done something very hard and very good. One significant exception to this rule appears to be sex and dating related sites, although the market is very saturated right now so a really significant twist or a really narrow focus within that space is required.

3. Don’t copy someone else’s wildly successful application and think you can add one little twist. It has got to be a really big "oh my God" twist.

This is perhaps the biggest crime of all. Its not even the me too apps. It’s the me three, or me fifty apps. It’s insane. Yes we all will have competition but why walk directly up to the shotgun and demand they pull the trigger.

The DOs

1. Do become a domain expert in some domain.
If you are a computer science/programming oriented entrepreneur, there is a great temptation to try to solve general broad problems. This is because we tend to have no other skills. If you study programming that *is* a skill, but you need to marry that with some understanding of some real world issue. I strongly suggest finding an activity that has a problem and learning inside and out about the people that do that activity and what real issues they have that you may be able to solve.

2. Do solve a real world problem that requires solving an interesting computer science problem.
This may seem like a bit of a contradiction with Do #1. Really they can of course go hand in hand and should. But if you have an interesting solution to a general problem you may be able to do it in a broad general domain instead of a narrower one.

3. Do solve a real world problem that requires some unpleasant engineering.
This is where most real software companies make their bread and butter. Its doing things that are just ugly. This really breaks down into two categories. One would be something like BuzzWord, the word processor that was recently acquired by Adobe that is written in Flash. It’s a really general problem and they are not solving any deep mysteries. But creating something that has never been done that is clearly hard on a new platform provides some level of competitive defense.

The second category is where the engineering is tedious. Plaxo is a good example of this. They have figured out how to sync with a whole bunch of contact managers. It means staying up on data type changes in the products they connect to and generally doing stuff that no self-respecting software engineer enjoys doing. That kind of stuff is very defensible and often very valuable. Plaxo may be a poor example of making something very valuable, but there are rumors of acquisition talks. I mainly chose them as an example because everybody knows them. But there are lots of software companies making real money doing this.

4. Do build a tool that helps people create things.
Tools that help people create stuff are worth money to people that make money creating things. If you can substantially increase someone’s productivity it is worth real money to them and therefore to you.

5. Do build a tool that helps people manage valuable resources.
This is really an extension of #4. Helping people manage time or money for example is very useful. That does not mean people will pay for a calendar program, but they may pay for a project manager. Or they may pay for something like WebEx, which is an online meeting tool that helps people avoid travel. It is often deployed for helping companies sell without having to meet in person. Yes it’s a communications app but it solves a distinct real world problem that helps people save time and make money. And it is not dependent on being viral.

6. Do build a tool that helps people sell something.
If you are selling things, particularly real world physical stuff, you can make money. It is not the most glamorous activity, but it does work. In order to get to real scale though, it is helpful to not be in the actual business of picking, packing and shipping. Etsy is a great example of this, as is PayPal. Needless to say this is hard because lots of people understand that if you want to make money, standing someplace where people are actually likely to give you money is a very good thing. But taking money from people (legally) will never go out of style.

7. Do Get lucky.
No matter what, obviously some luck is involved. But the key here is to get yourself in the ballpark. Doing some me too mashup widget is not likely to get you there.


In summary, if you start out internalizing the principles I have presented above and you start iterating and refining ideas as I suggest in the how to generate ideas article, I believe you can indeed come up with something that is unique and valuable. You may even end up with some idea that will work despite breaking one of the rules. But its like my old english teacher used to say, it is OK to break the rules, but the key is you have to know the rules before you can break them.