FacebookTwitter

The Hacker News Algorithm Is Extremely Fair

Summary – Hacker News Finds The Best Stories

Last weekend an article of mine was the top story on Hacker News and Reddit (r/programming) concurrently for the entire day. It was the first time I had seen an article “go viral”, and there were many unexpected aspects that I never anticipated including my server crashing completely. In this article I discuss how my post arrived at the top and why the Hacker News ranking algorithm is extremely fair.

Hacker News Viral

My Google Analytics while the article was viral showing 336 visitors from Hacker News

Surprisingly, my article did not reach the top of Hacker News until five hours after I submitted. It took over two hours for it to reach the top of Reddit (r/programming), and one of the strangest thing of all was the article appeared to be a complete failure in the first hour.

On the technical side, my WordPress website is hosted on a Microsoft Azure server which was set to scale. The server failed at three different inflection points, and later in the article I will show a behind the scenes look at what went wrong on the server side.

Azure

Microsoft Azure server – showing the dramatic jump in traffic

Hour One – No Signs Of Viral Article

I only shared the article in two places which were Hacker News and Reddit. I figured I would let the market decide if my content was worth reading or not. My Google Analytics showed there were roughly five to ten people on the site concurrently in the first thirty minutes.

After refreshing the Reddit thread, I noticed the article had a comment and was only 43% up-voted. The comment said that I was an idiot who was “butt-hurt” over the software hiring process. He added, “Tell us something we don’t know.” His comment was trolling me pretty hard, and I decided I better just close all my windows and step away from the computer. My wife asked me how the article did, and I told her it went very badly, and people don’t like the article for some reason.

Hour Two – Small Band of Redditors Join Me

After about an hour I summoned the courage to look at the Google Analytics again. It showed there were about 15 people from Reddit on my post now. It seemed while the post did not get many up-votes, there was a large comment section. The guy who called me an idiot had his comment down-voted many times now, and there were a few comments that my article was pretty cool.

Next, my article was on the top of r/programming. It seemed that people who read the full article were enjoying it. The Google Analytics showed 70, 100, 120, 170, 220, and up to 310 people on the site. A few people reported there was some caching issues with my article. Someone commented my website was getting hugged, referring to the Reddit hug of death.

Google Analytics shows about 40,000 sessions for the article on Sunday

Google Analytics shows 40,587 sessions for the article on Sunday

Hour Three – Signs of Life From Hacker News

The strangest part about the traffic from Hacker News was that there would be about 10 people coming from Hacker News, and then it would drop off to zero people. My article was very old on Hacker News now, and I had given up all hope of it doing well there.

However, the conversation on Reddit began to get a bit stale and I realized other people were submitting my article to Hacker News. Somehow the Hacker News algorithm realized that people from all over the world were submitting my article to the site. There was an entire army of Redditors demanding a discussion forum for my article. Suddenly, there were over 200 people from Hacker News on the site. I honestly thought it was some typo in Google Analytics. At this exact moment my server crashed.

Hour Four – The Server Crashes

My server began returning generic 500 errors and I had no idea of the cause. I had Microsoft Azure set to scale, so I thought they would manage everything for me in times of high traffic. My story was currently at the top of Hacker News and Reddit while the link was broken! I thought for sure the article would be down-voted instantly, but it remained number one in both places.

I tried restarting the server. It came back fairly quickly, but it still seemed to be jammed. My next thought was to check if I had paid my bills properly for the server and if I could upgrade some component of the server. I was presented with the following options:

Microsoft Azure Plans - Nine different pricing packages to choose from and I needed to make a decision quick

Microsoft Azure Plans – Nine different pricing packages to choose from and I needed to make a decision quick

I tried to figure out what I needed. Was it more RAM, more cores, more instances, more storage, or more slots? As soon as I clicked save on a higher priced plan, the traffic magically came back to the server. Every 10 seconds, another 50 or so people came onto the site, till it reached a peak of 800 people on the site.

Later I learned that I also needed to upgrade my backend MySQL database, and I added WordPress caching on the front end so that pages would load faster.

Conclusions – Hacker News Is Extremely Fair

It took all those Redditors submitting my article for it to reach the top. It is not supposed to be easy to have the top story on Hacker News. You need great content, a catchy title, newsworthy information, and some luck. The site does an excellent job of finding the best stories.

I would like to thank all those people who up-voted the story from Hacker News, and Reddit.

Did you find this story interesting? Leave me a comment below!

Submit a Comment

Your email address will not be published. Required fields are marked *