Gary's Longer Rants

No statistical evidence regarding amazon here.

Previous title: Less than .02366865 probability that Amazon does not censor 1-star reviews? This is stored here in lieu of deleting it totally because I don't like the idea of removing posts people have commented on, even posts I don't think have interest or value. [Further update: As I originally mentioned in the piece, the statistical calc was "for our amusement." I didn't take it very seriously and had no idea it would get this much attention. The (very appropriate) criticism emerged in discussions that at least some of the 10 Amazon reviews I was counting as the basis for the stats were undoubtedly the result of the original reddit attention to the Brian Powell issue, invalidating the calcs. I had explicitly noted that possibility in the original piece, but people didn't apparently didn't notice that very much and still took it seriously. As the discussion went on someone suggested that as few as 2 of the posts might have been legit. So I redid the calcs with 2, resulting in the .02366865 result in now the title: still statistically significant, and I did take it more seriously at that point. But now I'm trying to confirm that there actually were 2 "legit" reviews of the book in question on the day in question on Amazon. So far, one of the reviews I thought was likely to be independent has turned out to be in response to the reddit post. It's legit in the sense that the reviewer really did look at the book and write intelligently about it, but he did so in response to the reddit attention, which invalidates it from a statistical point of view. I have a couple more queries out there and if it does turn out that there are 2 independent reviews, I'll note it here. (If YOU wrote one on March 30 for The Final Theory, let me know.) For now, however, my feeling is that this was a not-very-serious and not-very-careful thing that got too much attention. Move on, there's nothing happening here.] [Update: Since this piece was posted there has been a fair amount of discussion about the calculations. I was really doing the most pessimistic possible calculation originally, since the actual number was more for my amusement than anything else, and I wasn't going to the trouble of actively screening out Amazon reviews that were very likely (or certainly) posted due to the attention this issue received on reddit. The probability was .0000000074279, and that number was in the title of this post. But since then there has been a lot of community input and I think the calculation should be taken more seriously. The most optimistic (from Amazon's point of view) version of the calculation still arrives at a probability of .02366864 -- which is still statistically significant. Accordingly, I've changed the title of the piece, but other than that and the updates in bold below, the piece is the same. Partly because I just don't want to spend more time on it, and partly due to a possibly misguided sense of "journalistic integrity". I may at some point right ANOTHER piece covering the same material and link to it from here. Also, if you are seriously interested in this, I strongly recommend that you read the discussion in the comments to get an idea of how seriously to take this result.] This morning I noticed this story by Brian Powell on reddit in which he contends that Amazon rejects one-star reviews. Brian wrote a one-star review for a popular science book. The book, called The Final Theory, apparently claims claims that most physics prior its publication was wrong, and that it advances the first overall theory of physics to be correct. As a grad student in physics, Brian unsurprisingly took exception to many of the book's assertions. His review was rejected repeatedly for different reasons. Finally he gave up in frustration, although at one point it was up on Amazon for two hours before being removed. After reading the reviewer's story I decided to investigate for myself. As of this morning (March 31, 2006) there are a total of 62 reviews. A total of 18 of them gave one star; most of the rest gave five. The question is: is Amazon censoring the negative reviews, as Brian contends? On the surface it looks like there is a mix of different reviews (albeit oddly bipolar). First of all, it seems odd that the overall review Amazon gives the book is four stars given that 18 of the reviews give one-star. I added up the reviews and found there were 18 one's, 5 two's, 0 3's, 4 fours, and 35 fives. The average review according to the simple calculation any elementary school student would do turns out to be 3.3 stars. I understand that Amazon has to round the number because they show the review as a graphic, but they do round to half-stars, so why are they rounding 3.3 up to 4? Something seems fishy here. But let's look a little deeper. Every one of the 10 reviews posted on March 30 (yesterday) gave one star. Hmm. That seems a trifle odd given that there are only 8 previous one star reviews since July 22, 2003 when the first (five-star) review was posted, among the other 52 previous reviews. I don't believe it takes a background in statistics to get a hint that those numbers don't add up. But for our amusement, let's look at it from a statistical point of view. Brian says one of his reviews was visible for two hours, and then removed. So apparently there can be a brief period when a one-star review is visible, before it is removed. Of course, that's a policy that would lead to their being a tendency for one star-reviews to be unusually prevalent among the very newest reviews (in this case, it's 100% of the reviews since yesterday morning). Another possibility is that Amazon is responding to the high visibility of Brian's post on reddit by suddenly letting the bad reviews through. But let's assume that Amazon would claim nothing is wrong: there is no underlying pattern that might lead to a higher propensity of one-star reviews since yesterday. That would imply that the any special preponderance of one-star reviews since yesterday would be a mere coincidence. So let's assume it is, and see where that takes us. More specifically, let's test the assumption that whatever factors might result in one-star reviews being published for a book like this one are not different in the period since yesterday morning than they were in the previous period. Any propensity for recent one-star reviews would therefore be mere innocent coincidence. In the previous period, we had a total of 8 one-star reviews, out of 52 reviews: a proportion of 0.15384615. What is the probability that our assumption is true? Using the assumption that underlying probability of .15384615, we can easily calculate the probability that all 10 out of 10 reviews since yesterday might be one-star. It's the same calculation as calculating the probability that tossing a coin 10 times in a row would be heads. As most people know, the way to do that is to multiply .5*.5*.5*.5*.5*.5*.5*.5*.5*.5. This is also denoted with the expression .5^10 (".5 to the 10th power"). In our case, we need to calculate .15384615^10. The answer is .0000000074279. In other words, there is a .00000074279% chance that the unusual preponderance of one-star reviews is due to coincidence. Note: this is not a rigorous calculation. It's an estimate. For instance, one might point out from the earlier 52 reviews, we can only get an estimate of the proportion of one-star reviews that Amazon would usually let through for a book like this. The upshot is that our computed probability is also an estimate. That's OK. It is standard in statistics to reject an assumption when the probability of its being true is .05. .0000000074279 is a very long way from .05. In addition, there may be other explanations to the propensity of recent one-star reviews. For instance, they could all be due to the reddit posting. However, reading the reviews, this does not appear to be the case. Most of them give ample evidence of direct knowledge of the book and thoughtful criticism. [UPDATE: This is in response to the comments I'm getting that I ignored the possibility mentioned in the paragraph above. I didn't ignore it -- I explicitly discussed it. Right there. Above. See it? About two inches above this sentence? I also said above that the calc is "for our amusement". It is NOT rigorous. The thing that I'll be interested to see is how many of those reviews remain over time. The hypothesis is that they are like Brian's review that was up for two hours before it was taken down. Will these reviews STAY up? I'll post something here when we know.] [FURTHER UPDATE: Someone over at reddit notes that there are now 11 neg reviews, and guesses that between 3 and 9 of the one-star reviews are due to the reddit posting. As the most pessimistic case assume that there are only 2 legit ones. That gives a result of .15384615^2 or 0.02366864; still below the usual statistical limit of .05 that leads the rejection an assumption (null hypothesis). So, as a statistical test, the conclusion stands even under most pessimistic of the guesstimates that appear in the the harshest numerical criticism I've received. However, I do now wish I had counted the suspicious ones instead of forming a rough impression before doing my initial calculation. It's a nuisance that people are focusing on the extreme number I calculated, when it really isn't necessary to be that extreme for deriving the statistically valid conclusion, and it IS probable that at least a couple of those reviews were due to reddit. I'd change the calc now, but there are already a lot of links out there so I think I should just leave it. This isn't a wiki such that it's supposed to be improved under community input; it's a post and is therefore more subject to journalistic standards.] One other thing should be mentioned. Brian's story quotes Amazon's responses to his reviews. On the surface, they sound like they could reflect a legitimate policy. But as Brian points out, the standards Amazon expresses there don't seem to be enforced for positive reviews. But even if they are, any policy which has the effect of screening out negative reviews more than positive ones, even if it is consistently applied, must be considered very, very suspect. On a personal note, I have made a lot of use of Amazon's reviews. I have thought that Amazon's reviews are one of the great Internet resources. But recently I've occasionally felt suspicion about them. It's seemed that there were too many five-star reviews for items that haven't seemed that good, and the summary star rating has sometimes seemed higher than my informal guesstimate of the average of the published ratings. This is the first time I've done a real analysis. Unfortunately it seems to confirm my suspicions: unless I'm missing something, Amazon now has policies in place that slant their reviews toward the positive. It's not hard to figure out why they might want to make the items they sell look better than they are! But I am very disappointed because I have always thought that Amazon reviews had integrity; I thought that they had realized that by having integrity, they earn the trust of buyers, and can use that trust to build their business. It always seemed to me that that was the smart approach, as well as the Right Thing, and it seemed to be what they were doing. But the fact is that Amazon is now in such a dominant position that their dominance probably wouldn't be threatened just because they lost some integrity. But, losing some integrity could increase revenues, as long as that dominant position remained. So it is arguable that their underlying financial pressures are now different than they were in earlier times, when they were still rapidly growing. And in fact, it appears that they may be responding to their new situation by having a new policy toward reviews. I very much hope that that is not true, or that if it is, that it's a temporary thing that Jeff Bezos somehow isn't aware of and that will be fixed. Right now, however, I'm feeling pessimistic. I notice that Brian says that Barnes & Noble did accept his review. I'll bet you can guess where I'm going to go next time I am considering purchasing a book.

Instructions for Training to Exaustion

TRAINING TO EXHAUSTION WITH A PRE-CLASSIFIED EMAIL DATABASE

Here's how the training set is handled.

Start with an empty word probability database.

Consecutively classify each email in the training set. Switch between ham and spam back and forth.

If an email is classified correctly, ignore it.

When an email is encountered that is misclassified, update the probability database using the misclassified email, and continue processing the emails in sequence using the updated probability database, until the last email is reached.

Go back to the first email in the training set and start again.

The above is repeated until either: a) the very last email is properly classified, or b) there is one complete iteration through the system with no improvement in accuracy.

Notes:

1) The probability database is not recreated during each iteration. Rather, the counts that lead to the probabilities are further refined with each iteration.

2) Some emails are counted more than once. In practice this happens relatively rarely. Also this can be avoided, possibly leaving wrongly classified messages in the end.

3) Experiments have shown that it is useful to use a so-called security margin. This is an interval around the cutoff value which is forbidden for all messages. For example, spams may be required to be >.7 and hams <.2. In practice, almost all mails can be made to conform to this (if you don't allow for messages used more than once, a few might not conform). Then, when .5 is used as the cutoff in testing, messages more likely get scores not close to the cutoff and are more likely to be classified correctly.

4) In preliminary tests, this procedure has let to very good performance.

5) Using this procedure, we don't need to tune the filter for parameters such as finding the ideal spam/ham cutoff point. Rather that probability database is automatically adjusted to match the cutoff point(s) initially chosen by the software developer (see 3 above).

6) In actual use in an inline spam filter, the process would ideally iterate through the entire database of emails every time there is a misclassification. If resources are limited this process might be restricted to recent messages.

My Position on Business Method Patents

This piece discusses my personal position on software patents, using a recent Microsoft patent announcement as the jump-off point.

Update: Slashdot already reports solid prior art.

"Microsoft has won a patent for an instant messaging feature that notifies users when the person they are communicating with is typing a message " [News.com]

Another stupid, obvious patent. AOL's AIM and Apple's iChat software infringe.

On the other hand, a News.com article from 2002 says:

America Online has quietly secured a patent that could shake up the competitive landscape for instant messaging software.

The patent (6449344), originally filed in 1997, and granted in September this year, gives AOL instant messaging subsidiary ICQ rights as the inventor of the popular IM Internet application. The patent covers anything resembling a network that lets multiple IM users see when other people are present and then communicate with them.

So it looks like AOL can fight back if necessary (I don't know where that would leave Apple and others). Update: Slashdot already reports solid prior art.

This is why companies that are responsible to their shareholders have no choice but to get software patents -- so they can fight back if necessary. The problem isn't with companies that get software patents, the problem is the environment. The PTO is so mindless in its inability to distinguish obviousness that way too many software patents are a joke, and do indeed have huge potential to obstruct industry progress. And software patents should be 3 years in duration rather than 20 because software evolves so quickly.

I am in complete agreement with Jeff Bezos on these issues. He's only doing what he has to do to uphold his fiduciary responsibility to his shareholders when Amazon gets patents. Arguably, he could be sued by irate shareholders if he didn't get them if it meant that Amazon didn't have the bargaining chips it needs to deal with patent lawsuits from others. (And it would mean that.)

Bezos argues that the system should be changed, and so do I. But while it exists as it does, he's going to do what it requires. So am I.

For the reasons stated above, plus a desire not to have Microsoft copy our hard-won ideas, Transpose, too, has software patents pending that could be considered to be "business method" patents. My current thinking on the issue, if those patents are granted, is to give free license to implementations which are free or result in only small profits, but retain the freedom to fight back against Microsoft and other large companies. I have begun to explore possible ways to build such licenses into the patents so that they couldn't be revoked.

Software patents suck, but if you're trying to be responsible to people who depend on you to make the right decisions financially, you gotta do what you gotta do.

You may have noticed that while I speak positively of Jeff Bezos above, I have also criticized specific Amazon patents.

As I said, Bezos is doing what he gotta do. He has to get those patents when he can to fulfill his responsibilities to his shareholders. That doesn't mean those patents are valid, i.e. enforceable by law. And Bezos knows it. Validity can only be tested in court. And look at who Amazon has sued: Barnes and Noble, which has plenty of legal resources to enable the judge and jury to fairly decide whether the patent is valid. No small companies or open-source projects have been harassed by Amazon over their patents.

The next step would be to do what I have outlined above, and actually provide an automatic free license to small businesses and open-source projects. That would increase Amazon's good will in the business and open-source worlds, and thus be good business, without harming Amazon's ability to duke it out with the Barnes and Noble's of the world. In effect, he has done that by not suing such embodiments of ideas patented by Amazon. Unfortunately, he has left the door open so that Amazon can change to a more obnoxious strategy if it wants to in the future, so we can't rest easy.

We'll see if they correct this in the future. I hope so.

I'll report here as my thinking (and the thinking of my company), evolves with respect to these issues.

Kazaa, BitTorrent, and Central Server-Based Systems

"Kazaa backs plan that could spell an end to the days of free music:"

The world's most popular song-swapping network, Kazaa, has thrown its weight behind a plan to start billing song swappers for their music downloads.

The proposal, which could finally end the days of the free lunch for millions of music fans, has been put to big US record labels at the same time as a new legitimate version of the former file-swapping giant Napster is launched in the US.

The idea is to phase in a billing mechanism for peer to peer networks, such as Kazaa and Morpheus, that allow users to copy music directly from each other's hard drives. [The Age]

It's hard not to be quite skeptical of such a plan. The reason peer-to-peer systems have flourished is because it was the only way to get easy access to large numbers of music files. There was no legal way to get them, and hugely expensive central server-based systems couldn't take the risk of making illegal files available.

Now that the legal agreements with the major labels are falling into place, central server-based systems can serve them. So the main reason now for peer-to-peer networks to exist is that they enable files to be shared at no cost. Once they start charging, their purpose in life won't be so clear.

At one point during Napster's original heyday, one of the managers at the company looked toward the future and said that the real reason for a system like Napster was collaborative filtering -- using the large numbers of people involved in such a system to form implicit communities to make recommendations to each other. For instance, everyone can see what is in everyone else's library. If you find someone whose tastes are like yours, then you may benefit from sampling items in their collection that you haven't heard yet.

In the end, this manager was right about the value of collaborative filtering, but confused about what it meant for peer-to-peer networks. Systems like AudioScrobbler let you see what other people have in their collections, but without caring how you got the files. The files can come from a central server -- it makes no difference where they came from. So the concept that peer-to-peer file sharing is justified by collaborative filtering makes no sense.

In the end, the advantages of stable, ultra-high bandwidth central servers will make peer-to-peer file sharing moot, except for that percentage of the population that continues, for one reason or another to choose not to pay for music. But that population won't be served by Kazaa under Kazaa's new plan.

There is one niche where legal, peer-to-peer sharing of music files may have a role to play in the future. This is in the sharing of legitimately free music -- music from artists who release some material freely in order to get people to pay for other material, and from artist that choose to make a living from concert ticket sales or who do it as a hobby and don't make money at all.

File sharing mechanisms such a BitTorrent make file sharing both speedy and reliable by sharing the load among a large number of peers. If one computer goes offline while it is serving a file, it doesn't end the transfer. The transfer continues from other computers in the network. That makes peer-to-peer comparably reliable to central server-based systems. There is still a disadvantage to something like BitTorrent, because each end-user computer shares the load of sending out files. In a central server-based system, the server carries all the load, and so there is less load on end-user computers. So BitTorrent-type systems will only flourish where there is a counterbalancing advantage compared to central server-based systems.

And indeed there are disadvantages to central server-based systems, even for serving legal files. A central server-based system will need to pay for the hardware and bandwidth to serve potentially huge numbers of files to potentially huge numbers of people. It will need to make a very large amount of money just to pay the overhead, and then it will want to make a profit on top of that. So its needs and the needs of users who just want the best free music are different. Such a system will have to either charge a sizable flat fee to pay for the overhead, or it will be inevitably try to convince people to buy more music when they would have been just as happy listening to a higher proportion of free music. Users will, of course, be able to choose to listen to large amounts of free music anyway, but the environment set up by the server-based system -- the content, the way the user interface works, etc. -- won't be particularly friendly to that choice. It will be a little less comfortable for people who are oriented toward free music than such people would like.

A reliable peer-to-peer system (such as one based on BitTorrent) could be set up that would be a perfect match to the needs of people who enjoy and care about free music. It wouldn't have to pay for the overhead of a central server-based system, so it wouldn't need to try to convince people to buy music when they would be just as happy with free music, and it won't need to charge high subscription frees to pay for bandwidth and hardware.

In fact, it fairly likely that a service will emerge that serves the needs of people who care about free music, and that it will be a peer-to-peer service. Kazaa has the opportunity to fill that role, but it sounds like Kazaa is a little too envious right now of the profits being made by the iTunes Music Store, and that will be made by other such stores, to see their opportunity. That may change in time.

(Of course the above assumes that Kazaa would prefer not to continue to base its business on the sharing of files that aren't legally available for that purpose. The fact that they are making moves toward establishing relationships with the labels is a strong argument that, in fact, they would prefer to make that change.)

A test of the hypothesis laid out above: will we ever see Amazon.com or the iTunes Music Store treat the needs of people who care about free music with as much priority as they treat the needs of paying consumers?

First Longer Rant

Except, it isn't actually longer!