If you were running an online platform or a service wherein you deal with user generated content, you would often want to keep a track of links that your users post and the corresponding clicks on those. We, as webmasters, love analytics of any sort because it can be put to great use – to make better products and to make lives simpler. Trouble begins when we start doing it the wrong way.
Some WebEngage customers complained that website referrer based rules in targeting for surveys were not working when they specified referring links from Quora (#get-more-context-here). We investigated the problem and realized it was one of those bad engineering practices that people use while choosing to do URL redirects to external websites from their web applications.
What’s the wrong doing?
Quora redirects in such a way that the destination website doesn’t come to know what the original site referrer was. Take this Quora thread for example – http://www.quora.com/What-are-everyday-apps-that-use-cloud-computing. As you can see, the answer by Mat Ellis contains a link to Gigaom –
http://gigaom.com/2010/06/08/how-zynga-survived-farmville/. Quora overrides it and converts the link to
http://www.quora.com/_/redirect?url=http%3A%2F%2Fgigaom.com%2F2010%2F06%2F08%2Fhow-zynga-survived-farmville%2F&sig=4f01ab. Overriding links is absolutely okay.
Response Headers HTTP/1.1 200 OK Server: PasteWSGIServer/0.5 Python/2.7.2 Date: Thu, 19 Jan 2012 12:37:52 GMT Content-Type: text/html; charset=utf-8 Pragma: no-cache Cache-Control: no-cache Content-Encoding: gzip Content-Length: 135 Response Body <html> <head> <meta http-equiv="refresh" content="0; url=http://gigaom.com/2010/06/08/how-zynga-survived-farmville/"> </head> </html>
As you can see, Quora emits a response body for those requests. This means the actual redirect happened to Gigaom as a page refresh from within the browser when the current location was
http://www.quora.com/_/redirect... instead of the old school
302 browser redirects. Now, if the Gigaom site or any script on Gigaom’s pages wanted to know which URL’s are users on their site coming from, all they would get to know is the URL
http://www.quora.com/_/redirect... They would never ever come to know where the user actually came from, e.g. in this case it should have been
A simple mistake and such loss of precious information.
If Quora would have simply redirected to the destination site from their backend, there would have been no problems at all. We understand that at times you don’t have a choice but to redirect via an HTML response body to the destination site. In that case, you should do it the way Google does it for its search results (Oh, you know that Google tracks your search clicks, right?). They changed their redirection mechanism a bit since the launch of new suggest feature. As you keep searching on Google with their suggest feature on, you only change a fragment (portion after #) in the URL. Browsers (read User Agents) don’t add such anchors, as they are called, to HTTP request header called Referrer. To counter this, Google composes their redirect URLs in such a way that it gives you the original search query in the redirect URL. Based on this particular parameter a lot of products thrive – the biggest being Google Analytics which tells you about search queries that led to your website. The point is simple – preserve and pass on the context for downstream applications to work as expected.
For the uninitiated, WebEngage is an in-site short survey tool. We let you create surveys and display those on your website in a “targeted” manner – basically, we let you filter based on multiple things like visitors geography, first time visitors, pages on your website, user’s browser etc. One of the targeting parameters is Referring site (images below).
For users trying to specify Quora URLs in the referring site section, it wouldn’t work as expected. We found out why and hence this blog post.
As online content sharing platforms keep growing, user are being offered innovative ways to share information. It becomes all the more important for these platforms to realize that there are hundreds and thousands of applications which empowers their users and others in variety of ways. Every single engineering decision they make, affects not only their own application but also the ecosystem around it. Moreover, getting it right is no rocket science either. Just stick to basics; there’s no need to re-invent because Sir Tim Berners-Lee has done all the hard work for us decades ago.
We rest our case here.
Update (20th Jan, 2012): After an uproar on Hacker News and a massive support for the cause, someone finally asked this question on Quora – http://www.quora.com/Why-does-Quora-redirect-to-URLs-in-a-way-that-loses-the-original-referrer. We’ll update this post when an official reply comes from Quora.
Update (28th Jan, 2012): We are pleased to announce that Quora has fixed its redirection logic to the old school way. This means no more broken web. We truly appreciate this gesture from Quora and thank the developer community for standing behind us.