• info@helpingtesters.com
  • helpingtesters

Web Automation with Sikuli and Selenium

April 4, 2016 Selenium Tutorial
Sikuli, Selenium, Web Automation, website test

During web automation, you face scenarios where you need to interact with Desktop. We all know Selenium only automates web browser, so when the situation comes where you need to upload images/documents in a website you need to interact with Desktop. Since Selenium can’t help with it, we need to use Sikuli library.

Sikuli automates anything you see on the screen. It uses image recognition to identify and control GUI components. It is useful when there is no easy access to a GUI’s internal or source code

Sikuli Features Enhancing Web Automation Capability

  • Screenshot of items on which we want to click, drag, double click etc their screenshots is taken and stored. During sikuli script execution these screenshots are matched with an actual screen.  Whenever a match is found i.e. that element is identified and you can interact with that element.
  • Sikuli can automate desktop application.
  • Sikuli can do web automation i.e. Web-based application.
  • It can even help us to automate mobile application but a workaround is required which we will be covering in later posts.
  • Sikuli can be used with our java eclipse program and can help selenium tool to automate GUI applications.
  • FLASH applications, games etc application whose elements couldn’t be identified with selenium because no identification elements like id, class, XPath or any locator is present, can be easily web automation with sikuli.


Sikuli Steps to Automate active Desktop Application

  • For using sikuli Java 8 must be installed on your system 
  • Launch eclipse and make a new workspace for the project.
  • Create new java project with name “SikuliProject”.
  • Open Project with “SikuliProject”
  • Double click on save jar file “sikulixsetup-1.1.0” and select second option 2 i.e Pack second: I want to develop in java, Jython or other java aware scripting language using NetBeans, Eclipse or other IDE’s
  • New jar file with name “sikulizAPI” will be created  
  • Create new class in eclipse and name it “DemoSikuli.java”
  • Link the sikuli jar file with java project Java Project–> Open Properties–> Libraries–> Add External Jar–> Select Sikuli jar  
  • On Desktop create folder with name “Sikuli”, we will be automating this folder using sikuli API  
  • Open Snipping Tool which comes with windows and takes image of it and saves to location, in my case “D:\images\Sikuli.PNG”    
  • Copy paste below code in eclipse:



import org.sikuli.script.FindFailed;
import org.sikuli.script.Pattern; //This is class, which will focus on images
import org.sikuli.script.Screen; //This is a class, which will focus on the screen

public class DemoSikuli {

      public static void main(String args[]) throws FindFailed{
      Screen screen = new Screen();
      Pattern image = new Pattern(D:\\images\\Sikuli.PNG);
  • Screen object represents the screen which it would be automation.
  • Pattern Object identifies the images which we will be comparing on the screen.
  • Pattern object path of the image which we would be clicking.
  • Click method clicks on the center of rectangle box which we would receive after comparing pattern on a screen.
  • Now execute java program and check whether sikuli first compares pattern with images on Screen. Then if it is successful it will click on folder “Sikuli” and you can see mouse moving.
  • Similarly Sikuli provides other function to “right click”, “double click” & others.


Drawbacks of Sikuli

  • Too many images need to be saved in order sikuli to identify elements.
  • Comparison is not always successful between similar images.
  • If resolution changes images needs to be again saved to automate them.
  • Sikuli automates only images taken by desktop so won’t work for mobile automation, a workaround is covered in our next posts.

About the author

Vaibhav Singhal author

Vaibhav is in software testing since a decade. Worked with different testing techniques along with exploratory testing. Performed different regression, performance, api etc testing with tools and libraries like selenium, jmeter, loadrunner, protractor, rest assured, Katalon Appium and others.



August 2, 2016 at 6:59 am

Like Like


November 13, 2016 at 5:16 pm

Thank you for your topic. It is helpful for me.

alok Jha

November 26, 2016 at 10:14 pm

very useful to me thank u very much……………


October 6, 2018 at 2:48 pm

Hi, This is really good post. As you mentioned, there is post for Mobile testing with Sikuli. Can you please share the link for same.

Leave a Reply

Your email address will not be published.