For the uninitiated, WebEngage is a simple customer feedback and short survey tool for websites. You can read more about the tool on WebEngage.com. We have a cool feature inside WebEngage called “try a demo“. In brief, the feature let’s you see beforehand as to how the WebEngage Feedback tab and the Survey window will appear on your site once the integration is done.
Slightly ambitious, but if Google planned to use WebEngage, this is how it would have looked on their home page
We have received innumerable requests from developers asking us to disclose how we did it. The urge is to such an extent that there is whole discussion thread on Stack Overflow around this feature.
http://stackoverflow.com/questions/7849466/showing-a-demo-of-my-css-on-any-website
This feature is a huge hit. Not only with developers but also with people willing to use WebEngage on their websites. In most cases we have seen that an online demo precedes a sign-up for WebEngage. Rightly so, because you want to see how it looks on your site before you go ahead with the integration. We have built an entire online demo application on top of this feature which we share with our bigger prospects. Before you read further, give the demo a try on your website – demo.webengage.com
There are three components to make such a demo functional -
- A Javascript widget that works cross-domain on a third party website
- A Web crawling bot that can fetch responses from any public URL
- A Web page parser to sanitize the response (#2) and modify it by inserting the widget (#1)
<webengage license="your_license_code"> <script id="_webengage_script_tag" type="text/javascript"> (function(){ var _we = document.createElement('script'); _we.type = 'text/javascript'; _we.async = true; _we.src = "//widgets.webengage.com/js/widget/webengage-min-v-2.0.js"; var _sNode = document.getElementById('_webengage_script_tag'); _sNode.parentNode.insertBefore(_we, _sNode); })(); </script> </webengage>
The code sample above is what we give out to our customers. The idea is to do all communication from a third party website via JSON requests – a technique in which you create dynamic <script> tags on a third party site to fetch data in realtime. These kind of Javascript widgets can also qualify to be a bookmarklet. Once the static JS above loads, you’d see that we make some dynamic requests thereafter to display the feedback tab and the survey window.
With robust libraries like Apache’s HTTPClient (JAVA), cURL (PHP) and PycURL (Python) etc, it is easy enough to fetch responses from URL’s. Here’s a snippet of how we did it in JAVA
public Object getPageByURL(URI uri) { HttpClient client = new HttpClient(); client.getHostConfiguration().setHost(uri); GetMethod method = new GetMethod(uri.toString()); int statusCode = client.executeMethod(method); if (statusCode == HttpStatus.SC_OK) { //process the response } }
There are multiple things to do here.
First, you need to make sure that all the resources that are needed to render the page are downloaded from the right location. So, if the webpage has relative URL’s for CSS, JS, Images etc, the browser should be asked to fetch it from the right location. The simplest way to do it is to add a <base> tag in the <head> tag of the page. Underneath is how -
<base href="url_of_the_resource_that_was_fetched_through_the_bot">However, there is a small twist – if the site already has a <base> tag defined in its page, you have to make sure that you are not overwriting it.
Second, you got to add the JS widget code at the end of the page.
To do both of the above, we needed to parse the page so that we could add these at the right location. Think ugly regex patterns and string manipulation. We decided to do it the easy way and Tidy our html response. Once the response was sanitized by this framework, it was very easy for us to identify the nodes we wanted to insert the above mentioned code snippets. But, to our surprise, a whole majority of users started complaining about the demo not working accurately as their websites looked drastically different in our demo as compared to what it would look like when viewed in the browser directly.
We scratched our head and almost everywhere in the body to figure out that these sites themselves were the culprit. They had malformed or untidy html markup. Tidy was removing or adding code to make it look tidy! And then we realized that the browser and Tidy behaved very very differently for untidy markups. For example, while the browser is okay with 2 <body> tags in a html response, Tidy is not – it will eat one up. (Un)fortunately, the browser renders _as_per_user_expectations. And we had to do the same thing.
Sounds easy right. Don’t parse the page at all. Pass it on to the browser the way it is. Well, yes and no. Remember, we need to insert two code snippets as mentioned above? So, we wrote a sweet and simple html parser which was completely fault tolerant – as in, it knew that html coders are drunk fellas who can choose not the start with <html>, who can choose not to close any tag they wish and who can have a <head> tag inside the <body> tag!
It’s an entire package we wrote and not all of it could be shared here. Here’s some pseudo code from a subroutine that you might find interesting.
/** * core parsing function which returns a Map of html Node name * and a List of HTMLTag's within it. The HTMLTag data structure looks * like this - * * public class HTMLTag{ * int indexStart; * int indexEnd; * int indexStartTagName; * int indexEndTagName; * String tagName; * Map<String, HTMLAttributeValue> attributeValues = * new LinkedHashMap<String, HTMLAttributeValue>(); * } */ public static Map<String, List<HTMLTag>> parseHTML(char[] html){ Map<String, List<HTMLTag>> map = new LinkedHashMap<String, List<HTMLTag>>(); int i=0; while(i < html.length){ if(html[i]=='<'){ HTMLTag htmlTag = new HTMLTag(); htmlTag.setIndexStart(i); i++; //findToken is a recursive function which binary searches for the //start and close of a particular tag with certain prefix and suffix //(the third and fourth parameters). It returns a //Token object (int startIndex; int endIndex; String token;) Token tagToken = findToken(html, i, whiteSpaceChars, whiteSpaceChars); String tag = tagToken.token; if(html[tagToken.endIndex] != '>'){ //parse for all kind of tags here //and store the data in map }else{ i = tagToken.endIndex+1; } }else{ i++; } return map; }
We have made sure to give you a wow experience with our demo feature. Hope this post post gives you enough insights to build your own.
Stay tuned. We love you!
Superb stuff Avlesh… the idea behind the feature is sheer creativity
As for engineering, it works so very seamlessly. You guys are on a roll !
Thanks a lot Bala for those kind words.
How do you capture the website screenshots? Can you please share the technique?
That would most likely be our next post Sameer
. To give you a quick idea – we post the DOM (complete html markup) from the frontend and generate an image on our server via a shell utility which uses a webkit rendering engine.
Cool. Got the idea. The logic to capture and sanitize the HTML will remain the same as explained above. Also got a fair idea about the image conversion part. Thanks Avlesh. Your products and ideas are awesome, keep up the good work.
Thanks for those kind words Sameer.
If im not wrong, When google + was launched, they had tried this little trick for UI related feedback, right ?
Well, the screenshot feature is far more than just a “trick”. That said, WebEngage is more than just a feedback tool. Have a look at our contextual short surveys – http://webengage.com/#survey-tab, you’d love it.