This video shows how to build a macro with XClick and XMove. The video uses the desktop automation mode, but everything works the same in web automation, except that the screenshots are taken directly inside the web browser. Note that XClick and XMove commands can not be recorded like Selenium commands. Instead, you build these visual macros by adding one visual command after another.
The XClick, XMove and XType command need the RealUser XModules to be installed. And please note that the RealUser module requires n unlocked user desktop to simulate events - no real human user can work with a locked desktop either.
Example of a web app click that fails with the standard CLICK command, but works just fine with XCLICK. Note that the same locator is used in both cases.
This sounds great, so why would one use the classic Click or ClickAt command at all?
Well, the real user commands XClick and XMove have one drawback: As they simulate real user input, they require an unlocked user desktop to simulate events, and the browser window has to be in the foreground. This is just the same requirement that you would have for a real human operator - no human can click on something if the desktop has been locked. Here the classic click/clickAt commands wins: They work just fine with the browser in the background and/or hidden, as everything happens just inside the web browser.
So for standard websites we recommend to first test if the classic Click command works. If it works, great. And if not, you have XClick in your toolbox. It will work for sure!
Automating websites with canvas elements (e. g. drawing apps like Sketchpad.io and most online games) using the ClickAt command could work, but it is impossible to find a XPath or CSS locator for ClickAt inside the canvas element. The best solution for canvas test automation is to use Kantu's computer vision image search. It searches visually for the image (e.g. a button image) that you want to click. For more details about image-driven test automation, see the Visual UI Testing page. Kantu's computer vision plus XClick/XMove/XType can automate everything inside canvas elements. More about using images as locator in the next paragraph below.
The standard selenium IDE way to trigger clicks inside an iframe is to first use selectFrame to switch to the right frame and then send a click event to the element. However, finding the right iframe number and the right locator can be tricky. And to make things even more complex, on some websites the iframe order and/or locator can change with each page reload . But the good news is that there is a better way to click inside iframes: Use XClick (image). The short screencast below shows how this is done in case of an embedded Youtube:
The locator input tells the command the location on the browser viewport where to send the native click and move events.
|"Locator" for XClick/XMove||Comment|
|(xpath or ID)||XClick can work with a normal page element locator, exactly the same one as you would use for the standard Click or Type commands. In this case Kantu locates the element, runs element.getBoundingClientRect() and then sends a click event to the center of the rectangle.|
|image@conf.Level||XClick can use images as input! - this works exactly the same way as with other visual commands like visualAssert. This is the easiest method to locate places to click, as it works 100% image-driven. Kantu's built-in computer vision finds the image in the active vision area and then sends a click event to the center of the image.|
|x,y||The raw way to send clicks by using x/y coordinates is also available. This can be useful if you know or can calculate the exact x/y position. For example, you can first do an image search, then calculate the coordinates as offset to this initial position.|
If you only want to check for the presence of an image you can use visualAssert.
XClick can simulate all kinds of mouse clicks.
|Click event command||Comment|
|„“ (empty) or #left||Standard left mouse click|
|#doubleclick||Simulates double-click of the left mouse button|
|#ctrlclick||Simulates CONTROL-Click with the left mouse button|
|#shiftclick||Simulates SHIFT-Click with the left mouse button|
|#middle||Simulates click of the middle mouse button|
|#right||Simulates a mouse right-click. Typically this is used to bring up the context menu.|
XMove simulates mouse events. For simulating mouse clicks, use XClick.
|Click event command||Comment|
|„“ (empty) or #move||Simulates mouse move events. A move over an element also triggers the mouse over effect.|
|#up||Simulates mouse up event of the left mouse button|
|#down||Simulates mouse down event of the left mouse button|
The DemoXClick macro that ships with UI.Vision includes many XMove commands, too. It uses the command to move the mouse cursor when drawing a rectangle on the canvas element.
(This page is work in progress. You find a full example in the Demo-XClick that gets installed with Kantu.)
Relative clicks allow you to click on something relative to something else. The idea is always that you find something (easy to locate) and then click on a often changing, very small or not unique element next to it. Or you find a label and then click in the input box to the right/left/top/below of it. Using Relative Clicks helps to create very stable visual automation scripts.
Screenshot: Using relative clicks. You draw the green box around the area to find, and the pink box indicates the area to click. So Kantu only searches for the image inside the green box. If you want to use this feature, make sure to use an X... command that ends with "...Relative".
You can draw the green and pink boxes with any image editor, but it will be easiest to use the build-in editor. It is probably the most basic image editor in the world, as its only features are to draw green and pink boxes. But it does this well ;-)
If you draw the boxes with an external image editor such as Photoshop, Microsoft Paint or Snagit, make sure that the colors of the boxes are code #00FF00 for green and #FE1492 for pink. Also make sure that the frames are one solid color, and that the tools do not apply any shadow effects, color gradients or similar visual effects to the border. This could disturb the box-finding logic. Of course, if you draw the boxes with the Kantu editor, this is automatically taken care of.
Kantu for Chrome and Kantu for Firefox with the XRealUser module installed.
The ready-to-import-and-run source code of all demo macros can be found in the Github repository testmacros folder.
As inspiration of what can be with the XClick, XMove and XType commands, we compiled an interesting list of web automation feature requests that we received from users over the last months before the release of the RealUser XModule. Now, all of these tasks can automated with the RealUser XModule.
How do I press "Ctrl C" and "Ctrl V" for copy and paste? It says can't find element - I'm trying to target cells on an excel sheet in google sheets and they don't have unique identifiers so I need to target "by coordinates" and then submit copy and paste WITHOUT caring what element i'm "sending" to.
I was trying to automate actions at https://www.ibm.com/... (URL removed) and would click on main documentation navigation headings (on the left) but the system couldn't detect when the AJAX response finished so it could verify an element. In retrospect I think it wasn't detecting the frame I was accessing.
It can't detect bootstrap modal.
Didn't work in Google Slides: I tried to record changing color and it didn't record anything except opening Google Slides (when it was already open)
I wanted to record click/drag operations in screen space, not on page elements. Kantu does not appear to support this.
add either the ability to send control keys or run an extension
I want mouseMove, mouseDown, mouseUp, keyboard events and timeline to record use cases for debugging purposes
Selenium IDE failed on my tests with our website (SPA, multiple frames). The error message was "can't access dead object"
compatibility with arrow, command/control etc. keys
In the website I'm testing, most links are accessed by hovering over a menu and then a submenu. When re-running a recorded macro, it failed after a time-out period when the link did not appear (because it did not perform the hover).
Probably outside the scope of the program, I'm looking for something with position based clicking rather than dom targeting.
looked like it worked but didn't actually activate buttons within gmail app to delete emails. Nothing appeard in the Trash folder??
Add the possibility to handle the closing of the actual tab.
I needed to click on an icon extension, but it did not work
When the website is created with Angular JS, the widgets change dynamically so the IDs of the elements change and Kantu is not able to see the elements.
doubleclick not supported,
it doesnt work on calendars year navigation,
Don't record angular webs
I need it to simulate all mouse movements not just controls being clicked or text boxes being filled
All of this can be done now - so if one if these questions is similar to yours, and you are not sure how to solve it, please ask in our forum.
...then please contact us.