Predicting the future popularity of a web page

Balloons in Trafalgar Square
Creative Commons License photo: wili_hybrid

New Scientist reports this week that a new tool developed at HP Labs could potentially predict the popularity of a web page in 30 days time. Essentially they say that by looking at the rate at which a web page picks up views in the first few days can predict the subsequent popularity of the page 90% of the time. It doesn’t seem too radical an idea – after all the pages which are more popular in the first few days are likely to get bookmarked more, linked to more, higher place on Google, etc.

The research focused around the sites Digg and YouTube so it would be interesting to see how it could be applied to other sites. You can download the paper online at

On a similar note, I’ve found that I’ve been able to get some incredibly stunning useful information from the popularity of webpages on my site. For example, one of my posts about MSN Messenger downtime gets a lot of hits whenever MSN Messenger goes down. When the number of visitors for that page is significantly above normal, I know that MSN is actually down. If the number of visitors is normal, it’s typically just an issue with my connection or my local server. In fact, I’ve found this method much more reliable than using Microsoft’s own service status page for the Messenger service. Similarly, I found a huge spike in the number of visitors to my post on the possibility of VAT cuts straight after the recent pre-budget report. If only there was a way of exposing these statistics in a useful way!

Free Delivery on – Charges Scrapped

Creative Commons License photo: joiseyshowaa

In a move which should improve price transparency, Amazon has essentially scrapped the delivery charge on its website.

To qualify for free super saver delivery, you now only need to spend £5 rather than £15. Given very few things cost under £5, this isn’t a tough feat! For orders under £5, you’ll still have to pay a delivery charge which is likely to dwarf the cost of your purchase. One book costs £2.75+VAT to ship and an electronic item £5.88+VAT to ship. In most cases, you will find it cheaper to buy a really cheap item in order to get your order up to £5 to avoid paying delivery. Use the Filler Item Finder for this. For example take a pack of 8 Duracell AA batteries for £3, or get a cheap kids book for 60p which you can recycle or use as a stocking filler.

I hate hidden prices online so I think it’s a fantastic move. A few weeks back, a company tried to charge me P&P for a ticket. Fair enough you might say but I think it was £1 for an e-ticket (because it actually costs them £1 to send an email!) or £1.65 to pick it up from the box office in person (because they pay their box office staff about £1.65 per half minute). Ebay has a certain problem with really cheap items with extortionate shipping fees. In some ways, Amazon Marketplace has had the same issue in the past… I remember buying an SD card for about £1.50 and paying £4.50 for P&P. Anyway, great news.

Sociologists describe Facebook "Ambient Awareness"

As a blogger and an avid user of social networking and new forms of web-based communication, I find it absolutely fascinating how they are changing the ways in which we communicate and live.

| apple-command |
Creative Commons License photo: arquera

It’s just passed the second anniversary of the introduction of the Facebook “News Feed”. For people who don’t use Facebook, the News Feed keeps you up to date on what’s going on in your social circle: new photos, wall posts, relationship statuses, events and parties your friends are attending and so on. I must admit that when I first signed up to Facebook I found this really scary. To me, it seemed really strange that a) instead of email, people would send communique to me by writing it on my wall which is publicly readable and b) these “wall posts” and my conversations would sometimes appear on the front page of Facebook for some friends of mine, who would then be provided with a link to view our entire conversation. When the news feed launched two years ago, 750,000 students protested against its launch.

Since the launch of news feed, Facebook has grown from 15million active users to 100million active users. It’s now become an integral part of the site, and the upcoming profile redesign makes the feed even more prominent throughout the site.

After two years of news feed, sociologists now describe an “ambient awareness” of friends. There is a fascinating article over at the New York Times about this:

Social scientists have a name for this sort of incessant online contact. They call it “ambient awareness.” It is, they say, very much like being physically near someone and picking up on his mood through the little things he does — body language, sighs, stray comments — out of the corner of your eye.

Look Up
Creative Commons License photo: mrhayata

The article goes on to describe microblogging where people post small and short updates throughout the day of their activities (e.g. Twitter or Facebook)

For many people — particularly anyone over the age of 30 — the idea of describing your blow-by-blow activities in such detail is absurd. Why would you subject your friends to your daily minutiae? And conversely, how much of their trivia can you absorb? The growth of ambient intimacy can seem like modern narcissism taken to a new, supermetabolic extreme — the ultimate expression of a generation of celebrity-addled youths who believe their every utterance is fascinating and ought to be shared with the world.

The article goes on to describe further research. At the end, Thompson concludes:

This is the ultimate effect of the new awareness: It brings back the dynamics of small-town life, where everybody knows your business. Young people at college are the ones to experience this most viscerally, because, with more than 90 percent of their peers using Facebook, it is especially difficult for them to opt out.

Certainly I think it’ll take a bit more time to see how Facebook changes the dynamics of society in the short run. At the moment, it’s impact is very limited to people at college and university. Perhaps it’ll be a welcome thing in our celebrity culture – where Paris Hilton and Wayne Rooney seem to be more important than issues such as climate change and where we feel we know celebrities better than our next door neighbours. Perhaps it’s a cool that’ll lead to a more responsible society.

Enquire Within Upon Everything
Creative Commons License photo: adactio

It’s just like living in a village, where it’s actually hard to lie because everybody knows the truth already,” Tufekci said. “The current generation is never unconnected. They’re never losing touch with their friends. So we’re going back to a more normal place, historically.

Psychologists and sociologists spent years wondering how humanity would adjust to the anonymity of life in the city, the wrenching upheavals of mobile immigrant labor — a world of lonely people ripped from their social ties. We now have precisely the opposite problem. Indeed, our modern awareness tools reverse the original conceit of the Internet.

Certainly the subject of some interesting research. What’s more, browsing the internet on our mobile is still something most of us seldom do. Here in the UK, various mobile companies have been advertising free Facebook access on your phone and new phones such as Android and iPhone make mobile internet access something which is much more palatable. We’ve definitely got more of this coming our way.

Making Money Online (and a living?)

What no one ever tells you about blogging
Creative Commons License photo: andyp uk

Much has been written about people who’ve quit the day job and taken up blogging or maintaining web sites full time. The claim is that you can spend a bit of time setting it all up and updating it with new content every so often and earn a lot of money very easily. In this post, I want to analyse whether this is true in practice. Further on in this post are also a couple of bits of advice to anyone who wants to monetise their website.

How much do you want to earn?

The most important question regarding making money online is how much you want to make and whether you can make it.

If you wanted to make a living from websites or blogging, the absolute minimum income you’ll want would be to have income equal to the level of “relative poverty”. This is how much money you’ll need just for the basic neccessities. In the US, its $10,787 a year. In the UK, it’s £11,326 a year ($21,900).

For my fellow Brits: you can already see how the exchange rate is stacked against us and the high costs of living don’t help. It’s always recommended that your primary sources of income and expenditure are in the same currency so you aren’t at risk from exchange rates volatility. It’s a source of worry if you want to become a full-time blogger and you live outside the USA.

Anyway, that means earning £215 a week just for the basic essentials. As the current exchange rate, that means a minimum income of $420 per week.

Let’s say your primary source of income is advertising. And let’s assume an eCPM (Earnings from 1,000 page impressions) of $1. So you’ll need 420,000 page impressions per week, or 60,000 per day. And if you want to earn a decent living, you’ll need at least 100,000 impressions per day.

It’s just a rough back of the envelope calculation. Of course, eCPM depends on all kinds of factors such as the subject of your site, the source of visitors (e.g. Digg traffic gives poor eCPM) and the number of advertisers.

Blogging tools, at Nerja Parador...
Creative Commons License photo: Ben30

Hopefully, I’ve quickly demonstrated that making a living online isn’t easy. Perhaps the exchange rate will become more favourable to website publishers in the future. Perhaps by redesigning your site you can increase your eCPM. But you’ll need close to a six-digit number of page impressions every day if you’re going to stand a chance.

What skills do I need?

If you want to be a professional blogger or webmaster, there are a whole range of skills you’ll need.

First and foremost, you’ll need to be able to write and to enjoy writing. After all, people come to your site for content and you’ll need to be able to write good, coherent content regularly.

You’ll need all the technical skills: web design, HTML, some CSS. You’ll need to know how to use a CMS or blog software. You’ll need to be able to optimise your pages for search engines.

Running a webpage professionally means you’re running a business. You’ll have to know how to communicate with advertisers and readers, manage cashflow…

It’s certainly possible to have all these skills, but making money online certainly is not easy or a “lazy” way of making money as it is often claimed on sites across the internet.

But if you do want to give it a go, a few bits of advice:

  • Make sure you find the topic of your blog/website interesting – otherwise it won’t be fun!
  • Think supply and demand. If you can supply something that is scarce (not many other people supply it) and high in demand (lots of people want it), you’re onto a winner. The fact that you need technical skills to run a blog means that blogging attracts the technical types. That means there are billions of blogs on computers and technology. If you do want to make a living, you probably won’t make it as a technology blogger. Sorry!
  • Keep experimenting with different ad programmes, placements and formats to maximise your earnings. Sometimes, rather surprisingly, fewer ads will give higher earnings.
  • You’ll probably find AdSense earning reports very addictive and check them many times every day… don’t.

Anyway… I hope this isn’t too demoralising for anybody! I write this blog entry as a technology analyst and an economist rather than somebody who has expertise or experience trying to make a living online. But I think people do need to know that it’s not the easy and lazy way of making money that people make it out to be.

P2P Intelligence and Monetising Media

Creative Commons License photo: Alex // Berlin (Stay in Madrid)

One interesting element of writing this blog is seeing how old posts are viewed. For example, last year I wrote about bugs inside my LCD screen. Since then, the post has been getting a couple of hits every day as people find it through Google and other search engines.

Over the last 3 or 4 days, traffic to that individual page has increased ten-fold. This coincides with the warm weather in the UK over the last few days. So obviously the fact I’ve been getting so much traffic on that page indicates that a lot of people have had problems with bugs inside their computer monitors lately.

It’s certainly an interesting piece of information or a trend that I’ve discovered by looking through my logs.

And it looks like media companies are finding their own sources of intelligence by looking at downloads on peer-to-peer sites. According to The Economist, 20 times as many tracks are exchanged on peer-to-peer sites than through legitimate stores.

Of course, sharing music on peer-to-peer sites is illegal. P2P sharing of music is not restricted to country borders and there is an infinite supply of it (since it’s free). That makes it dead useful for the media companies to discover where music is popular. For example, they can see where the music is popular even if it hasn’t yet been released there. And by finding out what other kind of music fans are listening to, especially in the younger generation which uses P2P, record companies can better organise supporting artists for tours.

Red iPod Nano
Creative Commons License photo: Andrew*

The dilemma for the record companies is of course that P2P file sharing is illegal and that means they are very reluctant to use data from it.

There is of course a middle ground. For example, Deezer is a legitimate music service which combines freely-available streaming music with a social network. Deezer spreads well through viral methods such as MSN Messenger, blogs and forums. Deezer can use cookies to link together all the different songs and artists you like. And when music costs nothing upfront, people are more willing to listen to new music. That means Deezer knows the music that you’re interested in: iTunes only knows what music you’ve purchased from them (and most people probably don’t buy all their music from one place).

Google Adsense transformed content on the web by showing relevant advertising. Amazon has transformed the way many of us shop by analysing what we buy and suggesting we buy them together. Instead of fighting piracy, I wonder whether these two approaches could be combined to align the goals of ensuring artists are properly rewarded with allowing people to discover new music organically and virally.

Should Internet Access be a Fundamental Right?

ice ice baby
Creative Commons License photo: mugley

A few days ago I wrote about the G8’s proposed ACTA legislation which could have a big effect on the Internet Freedom. And today I stumbled upon an article over The Register at a proposed French amendment to the European Telephone Package which is an EU-wide piece of legislation. The lowdown:

France has suggested an amendment to the pan-European Telecoms Package, which would bar broadband access to anyone who persists in illegally downloading music or films.

Last month, the government of Nicolas Sarkozy insisted on a similar “three-strikes-and-you’re-out” scheme for France. Under a cross-industry agreement, ISPs would have to cut off access for up to a year for third-time offenders. Sarko believes “there is no reason that the internet should be a lawless zone”.

I’ve had a bit of a think about the consequences of what would happen if I had no internet access.

  • In the UK, the only way to apply to Higher Education is through the internet. Anyone barred from the internet would have higher education closed off to them. Student finance is also set up on the internet.
  • Aspe Bag Advertising
    Creative Commons License photo: LiveU4

    The vast majority of my assignments and research involve research on the internet. So even if I could have applied to higher education and the relevant funding, I wouldn’t have been able to do anything when I got there.

  • I do a lot of banking online and some of my bank accounts (the e-savers) can only be accessed online. Without access to the internet, I would not have access to my own money.
  • I do much of my shopping online. If I couldn’t, I’d have to go shopping on the high street. For me, that often involves having to travel a fair distance and prices are usually quite a bit higher on the high street.
  • I personally make use of Facebook to organise much of my social life and I know that is the norm for many students of around university age. Losing internet access would also have a big impact on what I do in the “real world”.
  • I listen to much of my music and watch a fair amount of television and video on the computer. Without the internet, I would no longer have access to this entertainment!
  • Anyone who makes money online or works from home may be unable to continue in their current job.

culture is not a crime
Creative Commons License photo: Dawn Endico

I’m sure that all of us could think of a hundred reasons why we couldn’t live without the internet but I’ve listed a couple of my main ones and I’m guessing that a lot of people will agree with me on those.

As the internet becomes a more integral part of every day life, access to the internet becomes more and more important to being able to participate within society. So today I put this question out to you guys: Should the access to the internet be a fundamental right? Is it fair to remove somebody’s internet access for breaking copyright laws?

Global Anti-Piracy Treaty – Pretty Worrying

Copyright Criminal
Creative Commons License photo: amidanyorai

If you haven’t already read about this, the new G8 Anti-Counterfeiting Trade Agreement (ACTA) may peak your interest. It’s an international law so it’ll affect people around the world. ComputerWeekly has an article.

If agreed and implemented, the framework will give law enforcement officials new powers to enter and search premises and to confiscate goods that infringe copyright.

Critics believe this could extend officials’ powers, enabling them to seize and search laptops and iPods for illegal downloads.

And according to New Scientist (subscription required):

The proposed treaty has progressed with remarkable speed by the standards of international law. Quietly proposed by the Bush administration in September 2007, it quickly gathered support from the European Union, Australia, New Zealand, Canada, Japan, Switzerland and Mexico.

ACTA would make it illegal not just to share copyrighted material, but to operate websites that index the locations of such material that people can download. It would also outlaw systems like BitTorrent or Gnutella that help users find files on “peer-to-peer” (P2P) networks of computers.

When the move was first mooted last November, French president Nicolas Sarkozy dubbed it “a decisive moment for the future of a civilised internet”.

Pirates in tawny waters
Creative Commons License photo: jmpznz

Now I’m a firm believer in intellectual property rights and fighting piracy. As an economist, I know that these laws are very important for technological progress and the economic well being of the whole country. In fact, some people argue that without them the Industrial Revolution would have never happened. So in principle, anything to protect property rights is a good piece of legislation.

However, from what I’ve seen of ACTA, it appears that it has some big impacts on our freedom. ACTA makes it illegal to operate websites that point people towards the downloads of illegal material. Does this mean Google won’t be allowed to operate? What about my own site – I don’t knowingly link to anything which is illegal or copyrighted but since I allow comments from readers, there is no way I can ensure that comments don’t link to copyrighted material. Am I ultimately responsible?

I believe the right to link is one of the fundamental freedoms of the internet and this law could be dangerous. And if P2P software such as BitTorrent would be considered illegal, why wouldn’t FTP and HTTP?

I’m also worried about this law making it illegal to download copyrighted material. ACTA means ISPs can disconnect people from the internet for downloading copyrighted material. Seeing as we now do more and more things on the internet – run businesses, set up social events, do our grocery shopping – it’s a big threat.

radio debian
Creative Commons License photo: aloshbennett

How can one be sure that a website is properly licensed to use copyrighted material? Are the internet radio stations we listen to licensed? Well, they say they are. But then AllOfMP3 said they were and they’ve been closed down.

The effect of this is that people won’t get their music or films from any of the small retailers – only from the big names such as Amazon and MSN that they trust. It’s harmful to competition, and that’s bad news for the economy.

Certainly something to keep an eye on and we need some public debate.

Social Capital and Social Networking

Colombia, the only risk is wanting to stay
Creative Commons License photo: *L*u*z*a*

There is some fascinating research from Michigan State University about the use of online social networking sites by college students and the effect on their social capital. For non-economists, social capital is one of the three types of capital or “wealth”. The other two types are physical capital (what you own e.g. a computer, machines, money) and human capital (skills, experience). The third type of capital is your social connections – or social capital.

The study found that Facebook allows people to keep in contact with communities more efficiently, hence increasing their social capital. The authors of the study wrote, “Social capital has been linked to a variety of positive social outcomes, such as better public health, lower crime rates, and more efficient financial markets.” In addition, there was a correlation between Facebook usage and psychological well-being.

Of course, Facebook started off as a social networking site for college students only. Hence, it encouraged people to develop their relationships with people within their local college communities and people who they know offline which is more useful social capital than that which would be gained through forums and sites such as Digg.

Creative Commons License photo: Kate_A

The study goes on to look at two different types of social capital: bonding (cementing networks of homogenous groups of people) and bridging (social networks of socially heterogeneous groups of people). The argument is that some types of social capital are bad: criminal gangs, racist and extremist groups have high levels of bonding capital. The study found that Facebook tends to generate the good type of social capital: bridging.

Perhaps my criticism of the study is how it can be shown that Facebook causes an increase in social capital. To me, it seems pretty likely also that people with greater social capital will be more likely to sign up to Facebook.

But it’s certainly an interesting study. There certainly hasn’t been much study in the general area but I think it’s an area which needs studying: just as the web allowed us to enhance our human capital by learning more about different kinds of things, the social web will allow us to enhance social capital.

The “social web” may have drastic implications for our society and economy.

Proof of Concept: Using Facebook as Identity on a Blog

Please accept my apologies for the third successive post about Facebook!

I wanted to show off a proof of concept I’ve developed of using Facebook as an identity system for your blog. The proof of concept doesn’t run on a real blog system – that is simply to keep it simple and because I didn’t want to spend a lot of time with the ins and outs of WordPress. I can see no reason why it couldn’t be adapted to integrate with WordPress/Drupal or whatever.

How does it work?

  • You’re on the blog page and you wish to make a comment.
  • You click on a link which takes you to the Facebook webpage.
  • If you haven’t before used your Facebook identity on this application before, you’ll be asked to give permission for it to access your personal information.
  • You are returned to the blog page, now logged in with your facebook identity.

Total development time was about 1 hour, took about 50 lines of code. If anybody is interested, I can open source the PHP source code.

As far as I can see, it follows the platform terms of use. The only information which my proof of concept has access to is your name and the URL of your Facebook avatar. This information is not stored except with your expressed consent (e.g. you post a comment).

Pop over to the proof of concept and feel free to login or to post a comment to try it out.

Emotion Visualisation: We Feel Fine and Moodstream

Two fantastic visualisations of emotion today which may serve as inspiration for some of art, writing or perhaps even webpage designs.

We Feel Fine

We Feel Fine

We Feel Fine scours the internet for human feelings every ten minutes. According to their website, they use sources “including LiveJournal, MSN Spaces, MySpace, Blogger, Flickr, Technorati, Feedster, Ice Rocket, and Google”. They then analysewhat is written in the blogs for “I feel” or “I am feeling” and do further analysis/crawling around the website to pick up information on the type of feeling which is being felt as well as the age and geographical location of the author.

The applet on the website then generates a “emotion cloud”. Try it out.

Getty Images Moodstream


Moodstream is a visual brainstorming tool. Choose from 6 presets or select your own from various parameters such as happy/sad, calm/lively, humourous/serious, nostalgic/contemporary, warm/cool. The applet finds music, images and videos and combines them to deliver that image. It’s designed to be a take you in inspired and unexpected directions.