WebEngage Notification JavaScript Callbacks – Building A Realtime Conversion Engine On Your Website

Note: Breaking news first – we recently released our enhanced integration code which now supports a wide variety of JavaScript callbacks. If you are using WebEngage on your website, we recommend upgrading to the new code.

If you are new to WebEngage, take a quick live demo first – you’d love it!

Last week, we posted about our new integration code and how the callbacks can be put to some great use to build a “listening post” on your website for customers submitting feedback. Take a look at that here. This post focuses on how you can build some insanely beautiful interactions on your website using properties and callbacks in our Notification JavaScript API.

JS API awesomeness in Notification product

Our Notification product is a simple push messaging tool for your website. It allows you to display offers, run promotions and pop system alerts on your site from the comfort of WebEngage Dashboard – without changing any code on your site! Moreover, these messages can be “targeted” based on a bunch of rules. E.g. you can choose to display an offer ONLY to users “who are currently on the cart page AND haven’t checked out in 60 seconds AND have added atleast goods worth $200 or more”. Here’s more on targeting capabilities. You love us already? Wait, we haven’t even gotten started yet 🙂

Notification is an out-of-the-box product wherein all UI/targeting level customizations can be applied from the WebEngage Dashboard. That said, using our brand new JavaScript API, you can do a whole lot more – making the notification content personalized, building real-time interactions with users on a particular offer/deal etc. Take a look at the samples below.

The low hanging fruits first …
By default, a notification doesn’t get displayed to the same user once (s)he has closed the notification or clicked on any link/call-to-action in the notification. Now, by setting a property forcedRender to true, you can override this behavior. Take a look at the code sample and documentation below for other such properties. Add the corresponding properties to your integration code and see the magic unfold!

// by default, a notification doesn't get displayed to the 
// user once (s)he has closed it or clicked on any link/call-to-action 
// button in the notification. set this property to true
// if you want to override this behavior
_weq["webengage.notification.forcedRender"] = true;
// the WebEngage integration code pops a notification on 
// your website based on preset rules in the dashboard. no
// invocation is needed; however, if you'd like to manually 
// invoke a notification, you should set defaultRender to false
_weq["webengage.notification.defaultRender"] = false;
// render a particular notification by specifiying its ID
// you can get the ID of a notification from your notifications
// list page in the dashboard. Also, a notification gets
// rendered based on rules in the targeting section; if you'd
// like to skip rule execution, you'll need to additionally
// set skipRules to true 
_weq["webengage.notification.skipRules"] = true;
_weq["webengage.notification.notificationId"] = "~10cb5811a";

Creating hyper personalized messages for your users
We support tokens, which essentially is a way to display dynamic content inside notifications. Take a look at the notification below, as seen in the notification editor.

You can use any number of [[tokens]] in the editable fields – title, description, call-to-action text and URL. Through our JavaScript API, you can replace these tokens with realized values are shown underneath.

// to replace the tokens with their corresponding values, you'll need to 
// create a JSON key/value pair for all the tokens present in your 
// notification (across fields)
_weq["webengage.notification.tokens"] = { 
  userName: "John Doe"
// let's make this truly dynamic by asking you for a name which can be 
// dynamically replaced inside the notification (shown in pic above). 
// Underneath is a code. Give it a try by clicking on the "try me" 
// button below.
$("a.try", $(".notification-tokens")).click(function(){
  var name = prompt("Please enter your name", "John Doe");
  if(name != null){
    name = (name == "") ? "John Doe" : name;
    _weq["webengage.notification.tokens"] = { 
      "userName": name
    // we don't want WebEngage widget to automatically render an 
    // applicable notification as per the rules set in the dashboard
    _weq["webengage.notification.defaultRender"] = false;
    _weq["webengage.notification.skipRules"] = true;
    // we want to show the notification everytime you do a "try now" - 
    // irrespective of whether you have closed the notification once or 
    // have clicked on the call-to-action button; by default we don't 
    // show the same notification to a user once an action has been 
    // taken; this let's us override the behavior
    _weq["webengage.notification.forcedRender"] = true;
    // this is THE notification we want to render on this page
    _weq["webengage.notification.notificationId"] = "3176124c";
    // finally, render the notification!
  return false;

Implementing custom targeting rules & callback events
Custom targeting is an interesting way to display a notification (an offer / discount / recommendation etc) based on application specific criterion. E.g. Let’s say, you sell insurances online and you’d like to push a “child education plan” to customers while they are using your product online. However, you’d like to display this message ONLY to customers who have a “family group insurance”, have “atleast 1 kid” and have an “annual family income of more than $60,000”. Sounds cool? Here’s how you create such a notification and set-up its targeting criteria in your dashboard:

/* Implementing Custom Rules */
// for the custom rules, as shown above, to take effect on your website
// you'd need to pass the actual values of parameters hasGroupInsurance, 
// numberOfKids and annualFamilyIncome for logged in users.
_weq['webengage.notification.ruleData'] = {
  hasGroupInsurance: true,
  numberOfKids: 2,
  annualFamilyIncome: 75000
// with just three lines of the code above on your website, you can now
// target your notifications based any or all of the above mentioned
// customer attributes. Click the "try this example" button to see how
// this works with token data as underneath
_weq['webengage.notification.tokens'] = {
  userName: "John",
  kidsCurrentAge: 2,
  genderText: "her"
/* Implementing Callback On Click Events */
// first, you need to enable WebEngage callbacks for your widget
_weq['webengage.enableCallbacks'] = true;
/* onOpen callback */
// binding onOpen callback for your notifications
// this gets triggered anytime a notification pops-up on your website
_weq['webengage.notification.onOpen'] = function(data){
  // fetching the clickstream information for current user
  var userIP = (data.activity.ip ? data.activity.ip : "Unknown");
  var estimatedUserGeo = data.activity.city + " (" + data.activity.country + ")";
  var browser = data.activity.browser + " (" + data.activity.browserVersion + ")";
  var platform = data.activity.platform;
  var URLofThePageOnWhichFeedbackWasSubmitted = data.activity.pageUrl;
  var titleOfThePageOnWhichFeedbackWasSubmitted = data.activity.pageTitle;
  var referrer = data.activity.referrer;
  // the notification which got clicked
  var notificationId = data.notificationId;
  var notificationTitle = data.title;
  // if you are using tokens in your notification, this array
  // will contain all of them and have their corresponding values/defaults etc
  var tokens = [];
  for var(i=0; i<data.tokenData.length; i++){
    var aToken = data.tokenData[i];
    // the identifier for each token. e.g [[userName]]
    var tokenName = aToken.name;
    // you can set default value for each token in the dashboard
    var tokenDefaultValue = aToken.defaultValue;
    // the value set for each token in real-time via the JS API
    var tokenRealValue = aToken.value;
/* onClick callback */
// binding onClick callback for notifications
// if you attach this listener, every time a notification gets clicked,
// this function will be invoked
_weq['webengage.notification.onClick'] = function(data){
  // other than all the properties mentioned above, you'll
  // get these extra properties in the click event - the URL
  // and the text on the link/call-to-action button that got clicked
  var actionLink = data.actionLink;
  var actionText = data.actionText;    
/* onClose callback */
// triggers everytime a user clicks on the close button of the notification
// the data object, passed to this method, is the same as above
_weq['webengage.notification.onClose'] = function(data){

Custom data & advanced reporting
Using custom data, you can pass more information related to the customers viewing a notification or clicking on the call-to-action button. That ways, you can dice-and-slice the clickthrough information in your reporting section based on all the data you passed. Underneath is a code snippet of how to pass custom data and below that is a screenshot from the reporting section on how can clickthroughs be filtered based on these properties. Awesome. No? 😉
/* Implementing Custom Data */
// you can add a JSON key-value map to this property
// when clicks get recorded, this data gets saved along with click
// in your reporting module (screenshot below), you can filter the
// clickthrough data based on these properties
// supported data types: string, date, boolean, integer
// you can add any number of fields depending upon your plan
_weq['webengage.customData'] = {
  underTrial: false,
  domain: "userwebsite.com",
  daysInUse: 10,
  email: "admin@userwebsite.com"

That’s not all! You can do much more with other options in the API. Check out the documentation for JavaScript API (version 4.0) – docs.webengage.com along with our REST API’s and Webhooks.

Hope you liked what you saw. Do share your feedback on the API. A similar post here on how to use the JS API for our feedback product.

Stay tuned. We love you!

WebEngage Feedback JavaScript Callbacks – Building A Realtime Customer Feedback Listening Post On Your Website

Note: Breaking news first – we recently released our enhanced integration code which now supports a wide variety of JavaScript callbacks. If you are using WebEngage on your website, we recommend upgrading to the new code.

If you are new to WebEngage, take a quick live demo first – you’d love it!

This is how our new integration code looks like:
<script id="_webengage_script_tag" type="text/javascript">
  var window._weq = _weq || {};
  _weq["webengage.licenseCode"] = "_LICENSE_CODE_";
    var _we = d.createElement('script');
    _we.type = 'text/javascript';
    _we.async = true;
    _we.src = (d.location.protocol == "https:" ? 
               "//ssl.widgets.webengage.com" : "//cdn.widgets.webengage.com"
              ) + "/js/widget/webengage-min-v-4.0.js";
    var _sNode = d.getElementById('_webengage_script_tag');
    _sNode.parentNode.insertBefore(_we, _sNode);

In the previous version of our code, initialization of the widget was wrapped inside an onReady method. We kept it as such because of the async nature of our code plus a need to efficiently manage separate configuration and settings for all the three products from a central place once all the remote resources finished loading.

We got rid of that functional approach in this version. Instead, we created a JSON data structure with predefined keys that you can use to assign properties and attach callbacks for different events in context of each of the products.

JS API awesomeness in Feedback product

Our Feedback product allows you to create and configure a simple feedback tab for your website. You can add custom fields (radio buttons, text-boxes, dropdowns, file upload fields etc), create logical fields which are category dependent, build email routing rules and do a whole lot more from the WebEngage Dashboard. You can read more about the our feedback product here. The product is an out-of-the-box solution for your website. However, with the new integration code, you can add some interesting callbacks on your website to enhance the experience of users submitting feedback on your website. Underneath are a few samples.

UI level customizations
You can choose the default alignment, colors etc of the feedback tab for your website from the dashboard itself. However, let’s say on one of the sections on your site, you’d like to change these properties. Now, with the JS API, it is possible to do so. Just add these two lines to your integration code script on those pages/sections and see the magic unfold ..
_weq["webengage.feedback.alignment"] = "left";          // or, "right"
_weq["webengage.feedback.backgroundColor"] = "#cc9900"; // hex code

Form data handling and callbacks
With the new API, not only can you pre-populate the feedback form for your signed-in users, but you can also set field level attributes (like make mandatory, move to hidden field, reduce category options etc). So, let’s say, on a certain section of your site, you only wanted to display 2 categories in the dropdown instead of all, you can do that with formData property as shown below.

The new integration code (API) also provides hooks for onOpen, onClose, onSubmit events thereby giving you the ability to implement awesome user experiences for customers submitting feedback on your website. So, you can now build something like this – offer a discount coupon code to users submitting feedback on the order confirmation page! Sounds cool? Read through the code and comments below:

/* use this initialization property to pre-populate feedback form w/ user data */
_weq['webengage.feedback.formData'] = [ 
  // array of JSON's for each field in the form
    // pre-populate the field with label "email"
    name: "Email",
    // set the value here
    value: "john@doe.com",
    // change the rendering mode of this field. other options are:
    // "readOnly" - to prevent the user from changing the value passed above
    // "hidden" - to pass as hidden variable without showing the
    // field to the user
    mode: "default"
    // pre-populate the field with label "Category"
    name: "Category",
    // overrides what was set in the dashboard
    isMandatory: true,
    // if you intend to show only a few categories on a certain page
    // use this property to specify the options you'd like to show
    // these categories should be pre-created in the dashboard first
    options: [
      "Billing Inquiries", 
      "Sales/pricing related query",
      "General feedback"
/* Callback events - the fun begins now! */
_weq['webengage.enableCallbacks'] = true;
// So, let's say you wanted your webpage to be made "aware" everytime
// any user is either opening the feedback tab or submitting a feedback.
// Underneath is how you can attach your callback functions to these events
/* Feedback onOpen callback - triggers everytime a user clicks on the */
/* feedback tab on your website */
_weq['webengage.feedback.onOpen'] = function(data){
      "User data: [" + 
        "IP - " + data.activity.ip + 
        "Browser - " + data.activity.browser + 
                     "(" + data.activity.browserVersion + ")" +
        "Platform - " + data.activity.platform +
      "]" +
      "Page data: [" +
        "URL - " + data.activity.pageUrl +
        "Title - " + data.activity.pageTitle +
/* Feedback onSubmit callback - triggers everytime a user submits */
/* feedback via the tab on your website */
_weq['webengage.feedback.onSubmit'] = function(data){
  // in addition to the data that you get inside the onOpen method
  // you also get data submitted for each of the form fields 
  // along with user's geography etc.
  // iterating over the fields and fetching user submitted data
  var userResponse = {};
  for(var i=0; i<data.fields.length; i++){
    var field = data.fields[i];
    // while building your feedback form, you can choose any
    // kind of field - textbox, commentbox, radio-buttons,
    // checkboxes, file upload etc.
    // depending upon the field type, we wrap the responses
    // submitted by a user with corresponding "class" key in
    // the field object. E.g. for the auto screenshot field, when
    // used by a user, we provide a URL of the corresponding image
    // thumbnail that gets generated. for a detailed response
    // object view, click the "view response data object" button below
    var fieldValue = "";
    // auto screenshot field
    if(field.value["@class"] == "snapshot"){
      fieldValue = field.value.thumbnailImageUrl;
    // fields which produce a list of responses
    // e.g. multi-select, checkboxes etc
    else if(field.value["@class"] == "list"){
      fieldValue = field.value.values.join(", ");
    // fields which produce a textual response
    // e.g. textboxes, comment boxes etc
    else if(field.value.text){
      fieldValue = field.value.text
    userResponse[field.label] = fieldValue;
  /* Additional Information */
  // along with data submitted in the feedback form, our
  // API also shares with you additional details to
  // provide more context around the submission.
  var feedbackSubmittedOn = new Date(data.activity.activityOn);
  var userIP = (data.activity.ip ? data.activity.ip : "Unknown");
  var estimatedUserGeo = data.activity.city + " (" + data.activity.country + ")";
  var browser = data.activity.browser + " (" + data.activity.browserVersion + ")";
  var platform = data.activity.platform;
  var URLofThePageOnWhichFeedbackWasSubmitted = data.activity.pageUrl;
  var titleOfThePageOnWhichFeedbackWasSubmitted = data.activity.pageTitle;
  /* Taking action on user submitted data */
  // at this point, you know what data did the user submit
  // for each of the fields - name, email, category etc.
  // so, e.g. if you wanted to redirect the user submitting a sales
  // inquiry to your lead generation page and pass his/her
  // email to that page, here's how you can do it:
  if(userResponse["Category"] == "Sales/pricing related query"){
    // automatically close the feedback tab first!
    // now, redirect the user to the lead page
    window.location.href = 
      "http://mywebsite.com/lead.html?email=" + userResponse["Email"];

Wait, that’s not all! You can do much more with other options in the API. Check out the documentation for JavaScript API (version 4.0) – docs.webengage.com along with our REST API’s and Webhooks.

Hope you liked what you saw. Do share your feedback on the API. Here’s a similar post on how you can use our JavaScript API with the Notification product to build a realtime conversion engine for your website.

Stay tuned. We love you!

A peek into WebEngage’s security layer – super cool use of Java annotations

Note: If you are new to attribute programming, we recommend giving Attribute based programming or Java Annotations a read first.

To begin with, let us show you a small code snippet from one of our controllers. The saveSurveyResponse method underneath gets called every time some user takes an in-site survey on a customer’s website.

 * This method is invoked everytime someone takes a survey on a customer's website.
 * It performs two main tasks -
 * 1. saves the user's response in our database
 * 2. refreshes the stats and data graphs for the corresponding survey
public ... saveSurveyResponse(...) throws IOException {
  //mundane code here to compute a response object "surveyResponseDto"
  //first save the response in database
  this.publisherBc.saveSurveyResponse(licenseCode, surveyResponseDto);
  //then, refresh all the analytics associated with this survey
  this.publisherBc.refreshSurveyStatusOnResponse(licenseCode, surveyId);

As you can see above, the code performs two major tasks – #1. save the response in database and #2. refresh the analytics associated with the corresponding survey. We pre-compute a lot of data graphs and collate the information to create a bunch of Maps, so that the stats for our customers can be presented in real-time, in its true essence. That’s a lot of computing work. The method refreshSurveyStatusOnResponse usually takes about half a second (and much more at times) to complete. We can’t keep the end user on a third party website waiting because (s)he took a simple survey and we got in to some crazy computing business!

Solution? Simple – make that method call asynchronous.
Yes, one would either use a batch process or make that call asynchronous. However, it is a much bigger problem to address. There are several such methods in any application stack which should be executed asynchronously to keep the user experience intact. E.g. in any action that needs to send out an email, the sending email part can easily be made asynchronous because SMTP relays can be painful at times leading to huge lag; in the process, it keeps your end user waiting for a response.

Implementation? We created a cool Java annotation called Asynch.
We first created an annotation interface called Asynch and then implemented a method interceptor to look for this annotation on the callee. If the annotation were present, we’d execute it further in a new thread. Snippet from the implementation below.

 * Defining the Asynch interface
public @interface Asynch {}
 * Implementation of the Asynch interface. Every method in our controllers
 * goes through this interceptor. If the Asynch annotation is present,
 * this implementation invokes a new Thread to execute the method. Simple!
public class AsynchInterceptor implements MethodInterceptor {
  public Object invoke(final MethodInvocation invocation) throws Throwable {
    Method method = invocation.getMethod();
    Annotation[] declaredAnnotations = method.getDeclaredAnnotations(); 
    if(declaredAnnotations != null && declaredAnnotations.length > 0) {
      for (Annotation annotation : declaredAnnotations) {
        if(annotation instanceof Asynch) {
          //start the requested task in a new thread and immediately
          //return back control to the caller
          new Thread(invocation.getMethod().getName()) {
            public void execute() {
          return null;
    return invocation.proceed();

Done? Just that much?
Oh yes, pretty much. Here’s how the declaration of heavyweight method, refreshSurveyStatusOnResponse, looks like –

 * So, earlier we had a simple method in our interface which we later
 * annotated with the Asynch @interface. Bang! The caller doesn't need
 * to worry about it now. This method (no matter who the caller is)
 * gets executed asynchronously. Ain't that awesome? 
public void refreshSurveyStatusOnResponse(String licenseCode, Integer surveyId);

How did we use annotations to build our security layer?
Now that we gave you a fair idea of how we are using Annotations to our advantage, let’s dive a bit deeper. You are about to see how we built our entire Authorization and Authentication stacks using Java Annotation.

We are security freaks. Be it the web layer or data exchange layer, access to all our code is protected. The caller of a method is denied entry to the method if it does not have right privileges. Underneath is a small snippet from one of our web layer controllers. This method is invoked when someone tries to edit a survey from the WebEngage dashboard.

 * This is a public method invoked via a URL on the site. Once a user on the site
 * tries to reach this method, the "rules" specified below (via annotations) are
 * evaluated. If it matches with the criterion specified for the UserAuth class,
 * the user is allowed an entry into the method; otherwise is shown the exit door!
 * In the specific example below, only "signed-in" users who are "authorized 
 * publishers" (our terminology for WebEngage customers) AND have access to 
 * "survey configuration" related features, are allowed entry into this method.
@UserAuth (
  userTypes = {
  publisherUserFeatures = {
public ... edit(...) throws IOException{

Pretty nice. Right? So just by annotating the edit method with UserAuth, we made sure that survey edit URL’s returns a sweet nothing to those who are not supposed to use those URL’s. Where’s the beauty? This piece of annotation is reusable; we use it in a variety of ways on pretty much all the code that needs to be protected behind the concepts of user and their corresponding roles.

Of-course there’s a lot of application specific code behind understanding the annotation UserAuth. However, that’s one time and we have managed to reuse it in a much more sophisticated manner inside our Business Layer as well. Take a look at this usage in one of the methods below:

 * This is our business component method for customers who intend to change
 * the styling of their surveys to match the CSS with their site's look and
 * feel [oh, we have a sexy CSS editor inside dashboard for them to do so ;)]
 * This is a nested annotation. A list of @Authorize methods can be specified
 * as rules. Each of them specifying the method to be called (for authorization)
 * and the argument to be passed to it. For the caller to get access, these
 * values should meet the AuthRules criteria. If it does not, an 
 * AuthorizationException is thrown.
  authRules = {
      method = "hasPublisherUserFeatureAccess", 
      sargs = {"$0", "SURVEY_STYLING"}
      method = "hasPublisherFeatureAccess", 
      sargs = {"$0", "WE_SURVEY_CUSTOM_CSS"}
public void saveSurveyStyleCss(
  Integer publisherId, 
  AdhocAttributeName cssAdhocAttributeName, 
  String css
) throws AuthorizationException;

This explains how powerful annotations can get. At WebEngage, we use them to the fullest. Hope this article helps you build some cool stuff with attributes. Do let us know!

Note: Use the Asynch annotation idea with care. Spawning new threads without being in control can be fatal. If you plan to use it, make sure the threads are fetched from a pre-created thread pool.

Stay tuned. We love you!

How not to do URL redirects (… the way Quora does) [Update: Quora has fixed the issue]

If you were running an online platform or a service wherein you deal with user generated content, you would often want to keep a track of links that your users post and the corresponding clicks on those. We, as webmasters, love analytics of any sort because it can be put to great use – to make better products and to make lives simpler. Trouble begins when we start doing it the wrong way.

Some WebEngage customers complained that website referrer based rules in targeting for surveys were not working when they specified referring links from Quora (#get-more-context-here). We investigated the problem and realized it was one of those bad engineering practices that people use while choosing to do URL redirects to external websites from their web applications.

What’s the wrong doing?
Quora redirects in such a way that the destination website doesn’t come to know what the original site referrer was. Take this Quora thread for example – http://www.quora.com/What-are-everyday-apps-that-use-cloud-computing. As you can see, the answer by Mat Ellis contains a link to Gigaom – http://gigaom.com/2010/06/08/how-zynga-survived-farmville/. Quora overrides it and converts the link to http://www.quora.com/_/redirect?url=http%3A%2F%2Fgigaom.com%2F2010%2F06%2F08%2Fhow-zynga-survived-farmville%2F&sig=4f01ab. Overriding links is absolutely okay.

This is where the problem starts. Underneath is a snippet of the response (header and body) sent by Quora for such redirect requests –

Response Headers
HTTP/1.1 200 OK
Server: PasteWSGIServer/0.5 Python/2.7.2
Date: Thu, 19 Jan 2012 12:37:52 GMT
Content-Type: text/html; charset=utf-8
Pragma: no-cache
Cache-Control: no-cache
Content-Encoding: gzip
Content-Length: 135
Response Body
    <meta http-equiv="refresh" 
     content="0; url=http://gigaom.com/2010/06/08/how-zynga-survived-farmville/">

As you can see, Quora emits a response body for those requests. This means the actual redirect happened to Gigaom as a page refresh from within the browser when the current location was http://www.quora.com/_/redirect... instead of the old school 302 browser redirects. Now, if the Gigaom site or any script on Gigaom’s pages wanted to know which URL’s are users on their site coming from, all they would get to know is the URL http://www.quora.com/_/redirect... They would never ever come to know where the user actually came from, e.g. in this case it should have been http://www.quora.com/What-are-everyday-apps-that-use-cloud-computing.

A simple mistake and such loss of precious information.

If Quora would have simply redirected to the destination site from their backend, there would have been no problems at all. We understand that at times you don’t have a choice but to redirect via an HTML response body to the destination site. In that case, you should do it the way Google does it for its search results (Oh, you know that Google tracks your search clicks, right?). They changed their redirection mechanism a bit since the launch of new suggest feature. As you keep searching on Google with their suggest feature on, you only change a fragment (portion after #) in the URL. Browsers (read User Agents) don’t add such anchors, as they are called, to HTTP request header called Referrer. To counter this, Google composes their redirect URLs in such a way that it gives you the original search query in the redirect URL. Based on this particular parameter a lot of products thrive – the biggest being Google Analytics which tells you about search queries that led to your website. The point is simple – preserve and pass on the context for downstream applications to work as expected.

Why are we complaining?

For the uninitiated, WebEngage is an in-site short survey tool. We let you create surveys and display those on your website in a “targeted” manner – basically, we let you filter based on multiple things like visitors geography, first time visitors, pages on your website, user’s browser etc. One of the targeting parameters is Referring site (images below).

Specifying referrer based targeting for a WebEngage survey

Short survey in action

For users trying to specify Quora URLs in the referring site section, it wouldn’t work as expected. We found out why and hence this blog post.

As online content sharing platforms keep growing, user are being offered innovative ways to share information. It becomes all the more important for these platforms to realize that there are hundreds and thousands of applications which empowers their users and others in variety of ways. Every single engineering decision they make, affects not only their own application but also the ecosystem around it. Moreover, getting it right is no rocket science either. Just stick to basics; there’s no need to re-invent because Sir Tim Berners-Lee has done all the hard work for us decades ago.

We rest our case here.

Update (20th Jan, 2012): After an uproar on Hacker News and a massive support for the cause, someone finally asked this question on Quora – http://www.quora.com/Why-does-Quora-redirect-to-URLs-in-a-way-that-loses-the-original-referrer. We’ll update this post when an official reply comes from Quora.

Update (28th Jan, 2012): We are pleased to announce that Quora has fixed its redirection logic to the old school way. This means no more broken web. We truly appreciate this gesture from Quora and thank the developer community for standing behind us.

Try a demo feature – why and how we did it

For the uninitiated, WebEngage is a simple customer feedback and short survey tool for websites. You can read more about the tool on WebEngage.com. We have a cool feature inside WebEngage called “try a demo“. In brief, the feature let’s you see beforehand as to how the WebEngage Feedback tab and the Survey window will appear on your site once the integration is done.

Slightly ambitious, but if Google planned to use WebEngage, this is how it would have looked on their home page

We have received innumerable requests from developers asking us to disclose how we did it. The urge is to such an extent that there is whole discussion thread on Stack Overflow around this feature.


This feature is a huge hit. Not only with developers but also with people willing to use WebEngage on their websites. In most cases we have seen that an online demo precedes a sign-up for WebEngage. Rightly so, because you want to see how it looks on your site before you go ahead with the integration. We have built an entire online demo application on top of this feature which we share with our bigger prospects. Before you read further, give the demo a try on your website – demo.webengage.com

How did we do it?
There are three components to make such a demo functional –
  1. A Javascript widget that works cross-domain on a third party website
  2. A Web crawling bot that can fetch responses from any public URL
  3. A Web page parser to sanitize the response (#2) and modify it by inserting the widget (#1)
1. The JS widget
<webengage license="your_license_code">
  <script id="_webengage_script_tag" type="text/javascript">
      var _we = document.createElement('script');       
      _we.type = 'text/javascript';       
      _we.async = true;       
      _we.src = "//widgets.webengage.com/js/widget/webengage-min-v-2.0.js";       
      var _sNode = document.getElementById('_webengage_script_tag');          
      _sNode.parentNode.insertBefore(_we, _sNode);     

The code sample above is what we give out to our customers. The idea is to do all communication from a third party website via JSON requests – a technique in which you create dynamic <script> tags on a third party site to fetch data in realtime. These kind of Javascript widgets can also qualify to be a bookmarklet. Once the static JS above loads, you’d see that we make some dynamic requests thereafter to display the feedback tab and the survey window.

2. The crawling bot

With robust libraries like Apache’s HTTPClient (JAVA), cURL (PHP) and PycURL (Python) etc, it is easy enough to fetch responses from URL’s. Here’s a snippet of how we did it in JAVA

public Object getPageByURL(URI uri) {
  HttpClient client = new HttpClient();
  GetMethod method = new GetMethod(uri.toString());
  int statusCode = client.executeMethod(method);
  if (statusCode == HttpStatus.SC_OK) {
    //process the response
3. Response processor

There are multiple things to do here.

First, you need to make sure that all the resources that are needed to render the page are downloaded from the right location. So, if the webpage has relative URL’s for CSS, JS, Images etc, the browser should be asked to fetch it from the right location. The simplest way to do it is to add a <base> tag in the <head> tag of the page. Underneath is how –

<base href="url_of_the_resource_that_was_fetched_through_the_bot">

However, there is a small twist – if the site already has a <base> tag defined in its page, you have to make sure that you are not overwriting it.

Second, you got to add the JS widget code at the end of the page.

To do both of the above, we needed to parse the page so that we could add these at the right location. Think ugly regex patterns and string manipulation. We decided to do it the easy way and Tidy our html response. Once the response was sanitized by this framework, it was very easy for us to identify the nodes we wanted to insert the above mentioned code snippets. But, to our surprise, a whole majority of users started complaining about the demo not working accurately as their websites looked drastically different in our demo as compared to what it would look like when viewed in the browser directly.

We scratched our head and almost everywhere in the body to figure out that these sites themselves were the culprit. They had malformed or untidy html markup. Tidy was removing or adding code to make it look tidy! And then we realized that the browser and Tidy behaved very very differently for untidy markups. For example, while the browser is okay with 2 <body> tags in a html response, Tidy is not – it will eat one up. (Un)fortunately, the browser renders _as_per_user_expectations. And we had to do the same thing.

Sounds easy right. Don’t parse the page at all. Pass it on to the browser the way it is. Well, yes and no. Remember, we need to insert two code snippets as mentioned above? So, we wrote a sweet and simple html parser which was completely fault tolerant – as in, it knew that html coders are drunk fellas who can choose not the start with <html>, who can choose not to close any tag they wish and who can have a <head> tag inside the <body> tag!

It’s an entire package we wrote and not all of it could be shared here. Here’s some pseudo code from a subroutine that you might find interesting.

 * core parsing function which returns a Map of html Node name
 * and a List of HTMLTag's within it. The HTMLTag data structure looks
 * like this -
 * public class HTMLTag{
 *   int indexStart;
 *   int indexEnd;
 *   int indexStartTagName;
 *   int indexEndTagName;
 *   String tagName;
 *   Map<String, HTMLAttributeValue> attributeValues = 
 *               new LinkedHashMap<String, HTMLAttributeValue>();
 * }
public static Map<String, List<HTMLTag>> parseHTML(char[] html){
  Map<String, List<HTMLTag>> map = new LinkedHashMap<String, List<HTMLTag>>();
  int i=0;
  while(i < html.length){
      HTMLTag htmlTag = new HTMLTag();
      //findToken is a recursive function which binary searches for the 
      //start and close of a particular tag with certain prefix and suffix
      //(the third and fourth parameters). It returns a
      //Token object (int startIndex; int endIndex; String token;) 
      Token tagToken = findToken(html, i, whiteSpaceChars, whiteSpaceChars);
      String tag = tagToken.token;
      if(html[tagToken.endIndex] != '>'){
        //parse for all kind of tags here
        //and store the data in map
        i = tagToken.endIndex+1;
  return map;

We have made sure to give you a wow experience with our demo feature. Hope this post post gives you enough insights to build your own.

Stay tuned. We love you!