Try a demo feature – why and how we did it

For the uninitiated, WebEngage is a simple customer feedback and short survey tool for websites. You can read more about the tool on We have a cool feature inside WebEngage called “try a demo“. In brief, the feature let’s you see beforehand as to how the WebEngage Feedback tab and the Survey window will appear on your site once the integration is done.

Slightly ambitious, but if Google planned to use WebEngage, this is how it would have looked on their home page

We have received innumerable requests from developers asking us to disclose how we did it. The urge is to such an extent that there is whole discussion thread on Stack Overflow around this feature.

This feature is a huge hit. Not only with developers but also with people willing to use WebEngage on their websites. In most cases we have seen that an online demo precedes a sign-up for WebEngage. Rightly so, because you want to see how it looks on your site before you go ahead with the integration. We have built an entire online demo application on top of this feature which we share with our bigger prospects. Before you read further, give the demo a try on your website –

How did we do it?
There are three components to make such a demo functional -
  1. A Javascript widget that works cross-domain on a third party website
  2. A Web crawling bot that can fetch responses from any public URL
  3. A Web page parser to sanitize the response (#2) and modify it by inserting the widget (#1)
1. The JS widget
<webengage license="your_license_code">
  <script id="_webengage_script_tag" type="text/javascript">
      var _we = document.createElement('script');       
      _we.type = 'text/javascript';       
      _we.async = true;       
      _we.src = "//";       
      var _sNode = document.getElementById('_webengage_script_tag');          
      _sNode.parentNode.insertBefore(_we, _sNode);     

The code sample above is what we give out to our customers. The idea is to do all communication from a third party website via JSON requests – a technique in which you create dynamic <script> tags on a third party site to fetch data in realtime. These kind of Javascript widgets can also qualify to be a bookmarklet. Once the static JS above loads, you’d see that we make some dynamic requests thereafter to display the feedback tab and the survey window.

2. The crawling bot

With robust libraries like Apache’s HTTPClient (JAVA), cURL (PHP) and PycURL (Python) etc, it is easy enough to fetch responses from URL’s. Here’s a snippet of how we did it in JAVA

public Object getPageByURL(URI uri) {
  HttpClient client = new HttpClient();
  GetMethod method = new GetMethod(uri.toString());
  int statusCode = client.executeMethod(method);
  if (statusCode == HttpStatus.SC_OK) {
    //process the response
3. Response processor

There are multiple things to do here.

First, you need to make sure that all the resources that are needed to render the page are downloaded from the right location. So, if the webpage has relative URL’s for CSS, JS, Images etc, the browser should be asked to fetch it from the right location. The simplest way to do it is to add a <base> tag in the <head> tag of the page. Underneath is how -

<base href="url_of_the_resource_that_was_fetched_through_the_bot">

However, there is a small twist – if the site already has a <base> tag defined in its page, you have to make sure that you are not overwriting it.

Second, you got to add the JS widget code at the end of the page.

To do both of the above, we needed to parse the page so that we could add these at the right location. Think ugly regex patterns and string manipulation. We decided to do it the easy way and Tidy our html response. Once the response was sanitized by this framework, it was very easy for us to identify the nodes we wanted to insert the above mentioned code snippets. But, to our surprise, a whole majority of users started complaining about the demo not working accurately as their websites looked drastically different in our demo as compared to what it would look like when viewed in the browser directly.

We scratched our head and almost everywhere in the body to figure out that these sites themselves were the culprit. They had malformed or untidy html markup. Tidy was removing or adding code to make it look tidy! And then we realized that the browser and Tidy behaved very very differently for untidy markups. For example, while the browser is okay with 2 <body> tags in a html response, Tidy is not – it will eat one up. (Un)fortunately, the browser renders _as_per_user_expectations. And we had to do the same thing.

Sounds easy right. Don’t parse the page at all. Pass it on to the browser the way it is. Well, yes and no. Remember, we need to insert two code snippets as mentioned above? So, we wrote a sweet and simple html parser which was completely fault tolerant – as in, it knew that html coders are drunk fellas who can choose not the start with <html>, who can choose not to close any tag they wish and who can have a <head> tag inside the <body> tag!

It’s an entire package we wrote and not all of it could be shared here. Here’s some pseudo code from a subroutine that you might find interesting.

 * core parsing function which returns a Map of html Node name
 * and a List of HTMLTag's within it. The HTMLTag data structure looks
 * like this -
 * public class HTMLTag{
 *   int indexStart;
 *   int indexEnd;
 *   int indexStartTagName;
 *   int indexEndTagName;
 *   String tagName;
 *   Map<String, HTMLAttributeValue> attributeValues = 
 *               new LinkedHashMap<String, HTMLAttributeValue>();
 * }
public static Map<String, List<HTMLTag>> parseHTML(char[] html){
  Map<String, List<HTMLTag>> map = new LinkedHashMap<String, List<HTMLTag>>();
  int i=0;
  while(i < html.length){
      HTMLTag htmlTag = new HTMLTag();
      //findToken is a recursive function which binary searches for the 
      //start and close of a particular tag with certain prefix and suffix
      //(the third and fourth parameters). It returns a
      //Token object (int startIndex; int endIndex; String token;) 
      Token tagToken = findToken(html, i, whiteSpaceChars, whiteSpaceChars);
      String tag = tagToken.token;
      if(html[tagToken.endIndex] != '>'){
        //parse for all kind of tags here
        //and store the data in map
        i = tagToken.endIndex+1;
  return map;

We have made sure to give you a wow experience with our demo feature. Hope this post post gives you enough insights to build your own.

Stay tuned. We love you!

8 responses to “Try a demo feature – why and how we did it

  1. Bala

    Superb stuff Avlesh… the idea behind the feature is sheer creativity
    As for engineering, it works so very seamlessly. You guys are on a roll !

  2. Sameer

    How do you capture the website screenshots? Can you please share the technique?

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">