Easy Integration of Sikuli with existing framework of selenium

Nowadays Selenium is widely used tool for automating web application. In real life, products are enterprise application, so application would be an integration of  Silverlight ,Java applets,Raphael java script, canvas objects,Javascripts,Jquery. We could not expect everything to get identified with selenium and can face challenges in automating the few of the above mentioned UI with selenium.

Sikuli is one of the easiest tool to learn and can be used to automate anything using image recognition technique.

Here I am trying to share my experience about automating an enterprise application with integration of selenium and SIkuli.

I  was involved in automating the BPM product (business process modeling ),I have developed a framework in selenium to test this product .our functional team has developed around 1000+ tests using this framework.

Few months back our product development team has introduced new functionality of browser based modeling in which model can be developed in browser itself. This functionality was developed using canvas objects and Raphael javascripts.Automating this functionality was very important for us. Unfortunately selenium was unable to identify it. I tried all the options from selenium webdriver but all the efforts were in vain. Then We tried with different tools like Watir, Watij , AUTOIT and white to identify (business process modeling )BPM canvas objects but unfortunately these tools are also failed to get it automated.

                                                                                                                                                                                                                                            

While evaluating few other tools for BPM automation, I come across Sikuli (http://www.sikuli.org/), it’s an intelligent tool for recognizing the Canvas UI. It’s an open source tool.

It automates the UI using image recognition, and this image recognition technique was finest fit for us to identify Canvas objects and other modeler components from browser based modeler written in Raphael Java scripts.

Sikuli permits us to write the modularize tests in Java. Using this feature of sikuli, I  have integrated it in our selenium framework written in JAva

Sikulli team has provided neat and precise documentation about tool from basic to advance level, which guided us wherever we stuck up at any point while scripting with this tool.

As I mentioned earlier our functional team was expert  in writing the test using my existing framework written for selenium java. And I don’t want to expose them to new automation tool for automating one functionality from our huge product.so the main challenge for me was

ü  To integrate the sikuli with existing framework of selenium

ü  Developing the new keywords for sikuli , in a same way that I followed for selenium keywords,so that for our selenium test writer approach would remain same for writing the tests.

ü  With a detailed feasibility study of sikuli we observed that sikuli has its limitation and sikuli nowhere sits near selenium, so I decided to keep Sikuli’s use restricted to only for the UI which couldn’t get automate with selenium and remaining stuff with selenium.

How to integrate sikuli with existing framework of selenium ?

Sikuli provides sikuli-script.jar.you can download it from sikuli site  as recommended on the download page, Import this jar is your java application, once you have this jar in your java application.

You can invoke sikuli api’s in your selenium framework by creating Screen objects instance.

protected static Screen sikuliObject = new Screen();

Following is one of the simple way to call selenium and sikuli api alternatively as per the need of the test.

in following code ,Line 2 to 6 is of selenium code and line 11 to 13 is of sikuli code.

public void dragModelElementInCanvas(String modelElement, int x,int y) {
String elementToclick = "//a[contains(.,'"+ modelElement + "')]";
WebElement clickabelModelElement = seleniumDriver.findElement(By.xpath(elementToclick));
if (clickabelModelElement != null) {
 Actions action = new Actions(driver);
 action.clickAndHold(clickabelModelElement).perform();
 sleep();
//If you have been observed that selenium is not able to identify the component (e.g canvas)where you would like to above dragged element.
   System.out.println("moving mouse to the location " + x + " : " + y);
   sikuliObject.mouseMove(new Pattern(getAbsolutePath()).targetOffset(x, y));
   System.out.println("simulating mouse up");
 sikuliObject.mouseUp(Button.LEFT);
 action.release().perform();
  } else
   throw new KeywordException(
     "Timeout recieved while finding element with Xpath:"
       + clickabelModelElement);
 }

Important part here is for testwriter whomsever is using your framework will not have to bother which underline tool you are using to automate the stuff. For e.g in above code selenium testwriter will call the method dragModelElementInCanvas ,where he/she will not realize which tool has been used to achieve the task.

Above is the just the reference code that I have documented here for the one who will be getting started withselenium and sikuli’s integration. In real life we have developed many classes where we are using selenium and sikuli Hand to hand. Post your doubts if you are facing any specific issues while integrating selenium and sikuli.i would like to share my experience as we already overcome many challenges that we faced in integration of two automation tools in our existing framework of selenium.

Happy integration !!