XClick (target, click type), XMove (target, mouse event)

The XClick and the XMove commands allows you to send real user mouse clicks to web page elements. The advantage of this approach is that the web app reacts 100% the same way as if a real human user executes this click. In contrast the standard Selenium IDE Click and ClickAt commands operate on the Javascript level inside the browser DOM, and can sometimesr require dedicated debugging and tweaking to make them work (e. g. finding the right locator) or they can fail completely on complex websites, as the website logic swallows the click event and does not send it to right element. XClick avoids all these problems by simulating a real user.


This video shows how to build a macro with XClick and XMove. The video uses the desktop automation mode, but everything works the same in web automation, except that the screenshots are taken directly inside the web browser. Note that XClick and XMove commands can not be recorded like Selenium commands. Instead, you build these visual macros by adding one visual command after another.

The XClick, XMove and XType command need the RealUser XModules to be installed. And please note that the RealUser module requires n unlocked user desktop to simulate events - no real human user can work with a locked desktop either.

What is the difference between XClick/XMove and the Selenium IDE Click and ClickAt commands?

The classic Click commands as used by UI.Vision RPA and any other Selenium IDE or browser automation tool do not simulate real mouse clicks! Instead they operate on the Javascript level inside the browser DOM. This the reason why these commands all first require a locator to identify a web page element, and then send a Javascript mouseclick event to this element. On simple websites this is often enough and works well. But on modern complex websites (e. g. Google Gmail, Google Spreadsheet, Microsoft Office 365, Facebook, Instagram,...) it is often difficult to find the right element locator in the first place. And even if you found the right locator, it can happen that the website logic swallows the Javascript click event event and it never reaches the target right element. In contrast the XClick and XMove commands avoid all these problems by simulating a real user click and mouse events.


Example of a web app click that fails with the standard CLICK command, but works just fine with XCLICK. Note that the same locator is used in both cases.

This sounds great, so why would one use the classic Click or ClickAt command at all?

The real-user commands XClick and XMove have one drawback: As they simulate real user input, they require an unlocked user desktop to simulate events, and the browser window has to be in the foreground. This is just the same requirement that you would have for a real human operator - no human can click on something if the desktop has been locked. Here the classic click/clickAt commands wins: They work just fine with the browser in the background and/or hidden, as everything happens just inside the web browser.

Multi-Monitor Support: Note that the the image search/OCR for XClick/XMove is always done on the main screen. So even if UI Vision runs e. g. on the 3rd monitor, the computer vision will search the main screen only. This means the browser or app that you want to automate with XClicks needs to be placed on the main screen. We plan to add multi-screen support for image search with a future update.

So for standard websites we recommend to first test if the classic Click command works. If it works, great. And if not, you have XClick in your toolbox. It will work for sure!

How to test canvas elements?

Automating websites with canvas elements (e. g. drawing apps like Sketchpad.io and most online games) using the ClickAt command could work, but it is impossible to find a XPath or CSS locator for ClickAt inside the canvas element. The best solution for canvas test automation is to use the computer vision image search of the RPA software. It searches visually for the image (e.g. a button image) that you want to click. For more details about image-driven test automation, see the Visual UI Testing page. RPA computer vision plus XClick/XMove/XType can automate everything inside canvas elements. More about using images as locator in the next paragraph below.

How to click on elements inside an iFrame?

The standard selenium IDE way to trigger clicks inside an iframe is to first use selectFrame to switch to the right frame and then send a click event to the element. However, finding the right iframe number and the right locator can be tricky. And to make things even more complex, on some websites the iframe order and/or locator can change with each page reload .

But the good news is that there is a better way to click inside iframes: Use XClick (image). The short screencast below shows how this is done in case of an embedded Youtube:

XClick and XMove "Locator" options - Computer vision, XPath or X/Y

The locator input tells the command the location on the browser viewport where to send the native click and move events.

"Locator" for XClick/XMove Comment
(xpath or ID) XClick can work with a normal page element locator, exactly the same one as you would use for the standard Click or Type commands. In this case UI.Vision RPA locates the element, runs element.getBoundingClientRect() and then sends a click event to the center of the rectangle.
image@conf.Level XClick can use images as input! - this works exactly the same way as with other visual commands like visualAssert. This is the easiest method to locate places to click, as it works 100% image-driven. UI.Vision RPA's built-in computer vision finds the image in the active vision area and then sends a click event to the center of the image.

If there is more than one image that matches the input image the best match is selected. But you can force UI.Vision to select another match above threshold with #. So to click the 3rd matching image use image_dpi_96.png#3 or image_dpi_96.png6@0.75#3. Once # is added, all matches above the confidence level are treated as equal and are sorted from top/left to bottom/right. Related forum posts: Top/left to Bottom/Right sorting explained and How to click all matching images.
ocr=text to click

ocr=[text to click] (exact match)

ocr=text to click@pos=X
XClick can read just like a human: When you use ocr=text, UI.Vision RPA takes a screenshot of the page, reads it (OCR) and then finds the text to click. Since this works on the rendered website image, it operates just like a human. Unlike with the HTML-based xpath text() command, there are no tricky xpath, iframe and Javascript issues to take care. The built-in computer vision of the RPA software finds the text in the active vision area and then sends a click event to the center of the text. This work with text inside images, videos or PDF documents, too.

The default text match mode is partial match: "ocr=day" matches "day" but also "today" or "yesterday's". For exact match use square brackets [text]. Exact text match is useful, for example, when automating a date control: There will be the numbers 8, 18, 28. With ocr=[8] you make sure that the number 8 is clicked, and not 18 or 28.
To click the X-th occurrence of a text string, use ocr=text@pos=X. The occurrences are counted from top left to bottom right. Another option to exclude some matches is to limit the search area.
x,y The raw way to send clicks by using x/y coordinates is also available. This can be useful if you know or can calculate the exact x/y position. For example, you can first do an image search, then calculate the coordinates as offset to this initial position. If you want to re-use x/y from a previous image search, you find them in ${!imagex} and ${!imagey}. Also, there are good tools to determine x/y manually. Note that in browser automation mode the XClick | 0,0 value is the top left of browser viewport. In desktop automation mode it is the top left of the main desktop.

If you only want to check for the presence of an image you can use visualAssert.

XClick Click types

XClick can simulate all kinds of mouse clicks.

Click event command Comment
„“ (empty) or #left Standard left mouse click
#doubleclick Simulates double-click of the left mouse button
#tripleclick Simulates triple-click of the left mouse button (related RPA tutorial video, starts at 2:50)
#ctrlclick Simulates CONTROL-Click with the left mouse button (related forum post)
#shiftclick Simulates SHIFT-Click with the left mouse button
#middle Simulates click of the middle mouse button
#right Simulates a mouse right-click. Typically this is used to bring up the context menu.

XMove Mouse events

XMove simulates mouse events. For simulating mouse clicks, use XClick.

Click event command Comment
„“ (empty) or #move Simulates mouse move events. A move over an element also triggers the mouse over effect.
#up Simulates mouse up event of the left mouse button
#down Simulates mouse down event of the left mouse button

The DemoXClick macro that ships with UI.Vision includes many XMove commands, too. It uses the command to move the mouse cursor when drawing a rectangle on the canvas element.

XMove Drag & Drop

XMove is used for dragging & droping items in the web browser and on the desktop. Drag and Drop is a sequence of the two mouse events XMove |...|#DOWN and XMove |...|#UP:


Drag and Drop with XMove. For details see this forum post: Drag & Drop with RPA software

XClickText

XClickText works like the existing "XClick | OCR=text" command with one difference: XClickText automatically retries the text search if the word is not found. It retries until the !timeout_wait limit is reached. The retries work exactly as with XClick, but for text search instead of image search.

Also, because of these retries, XClickText always uses the local XModule OCR engine. Online OCR would be to slow for this purpose. So XClickText always runs locally. For OCR with the hosted OCR engines (SaaS), please continue to use "XClick | ocr=".

XMoveText

XMoveText works the same as XClickText, but triggers a "mouse over" event instead of a click.

XClickTextRelative

XClickTextRelative is available for web automation and desktop automation. If you use it for desktop automation, see the Calibrate XClickTextRelative paragraph below. In web automation mode (OCR inside the web browser) you do need the calibration.

To get started with the command, we recommend that you run "DemoXClickTextRelative" to see the command in action. This macro is installed with Ui.Vision by default.

What does for example "#R5,-10" mean?
XClickTextRelative | Login#R5,-10 means that computer vision first finds the word "Login", then - from the center of this word - the RPA software moves the cursor 5 chars to the right and 10 chars down (Down because it is -10, Up would be +10).
The term "#R" has no special meaning, it is only the separator between anchor word and coordinates.


XClickTextRelative works similar to XClickRelative. But instead of an anchor image, it looks for an anchor WORD. Then, from this word, it goes X times left/right and Y times up/down.

Shift click from middle of the bounding box
Image: You can also use the ...TextRelative command to shift the click position from the default (center of the bounding box).

XMoveTextRelative

XMoveTextRelative works the same as XClickTextRelative, but triggers a "mouse over" event instead of a mouse click.

XClickTextRelative, XMoveTextRelative and OCRExtractbyTextRelative in DESKTOP mode

If you use these three commands in desktop mode AND you are on Windows AND you are using a HiDPI screen AND you are using a screen scaling that is different from 100% then (and only then) you need to calibrative the relative distance calculation used by the RPA software. If you are on a different system and use the calibration, nothing bad happens. It is just not needed then.

To calibrate, do this:
- Enter your screen scaling value in the box e. g. "200" for a 200% screen scaling.
- Press the "Calibrate XClickTextRelative" button. This also calibrates XMoveTextRelative and OCRExtractRelative.
- A desktop OCR preview window opens it. You can close it. The value in the OCRTEXTX box gets adjusted automatically by the calibration (if needed). For example, it might change from "6" to "13". The calibration is now done.

XClick Example

(This page is work in progress. You find a full example in the Demo-XClick that gets installed with the RPA software.)

Command Target Pattern/Text
open https://ui.vision/
XClick button.png #doubleclick
echo done

Computer vision: Relative Clicks with green and pink boxes

Relative clicks allow you to click on something relative to something else. The idea is always that you find something (easy to locate) and then click on an often changing, very small or not unique element next to it. Or you find a label and then click in the input box to the right/left/top/below of it. Using Relative Clicks helps to create very stable visual automation scripts.

Relative Clicking with the EXTRACT command...
Screenshot: Using relative clicks. You draw the green box around the area to find, and the pink box indicates the area to click. So UI.Vision RPA only searches for the image inside the green box. If you want to use this feature, make sure to use an X... command that ends with "...Relative".

The green and pink boxes need to be separate, not inside another
Screenshot: Make sure that the green and pink are not inside each other, that confuses the computer vision.


OCRExtractRelative is explained in this video. Creating the green/pink box works the same for all ....Relative commands. The green box is always the anchor image. The difference is the meaning of the pink box:
(1) For OCR, the pink box marks the text area to read.
(2) For visionLimitSearchAreaRelative the pink box marks the new search area for computer vision.
(3) For XClickRelative/XMoveRelative the middle (center) of the pink box marks the point to click.

You can draw the green and pink boxes with any image editor, but it will be easiest to use the build-in editor. It is probably the most basic image editor in the world, as its only features are to draw green and pink box (XClickRelative forum post).

If you draw the boxes with an external image editor such as Photoshop, Microsoft Paint or Snagit, make sure that the colors of the boxes are code #00FF00 for green and #FE1492 for pink. Also make sure that the frames are one solid color, and that the tools do not apply any shadow effects, color gradients or similar visual effects to the border. This could disturb the box-finding logic. Of course, if you draw the boxes with the UI.Vision RPA editor, this is automatically taken care of.

Works in

UI.Vision RPA for Chrome, Edge and UI.Vision RPA for Firefox with the XRealUser module installed.

The XType, XClick and XMove need the RealUser Simulation XModule to be installed.

XMouseWheel: Simulate mouse scrolling events

XMouseWheel simulates mouse wheel scroll events. A negative value means to scroll down/zoom in and a positive value is used to scroll up or zoom out:


How to simulate mouse wheel scrolling with XMouseWheel: The video shows how to automate zooming in and out on Google maps. First we "mouse over" the city of Oxford with XMove and then we zoom in (-200) and out again (+200). We use two XMouseWheel commands for each scroll direction simply to slow the scrolling/zooming down a bit, so it is easier to see.

Related Demo Macros

Demo-XClick

The ready-to-import-and-run source code of all demo macros can be found in the Open-Source RPA software Github repository.

See also

XMove, XType, XModules User Manual

Real User's problem where Real User simulation is the answer

As inspiration of what can be with the XClick, XMove and XType commands, we compiled an interesting list of web automation feature requests that we received from users over the last months before the release of the RealUser XModule. Now, all of these tasks can automated with the RealUser XModule.

How do I press "Ctrl C" and "Ctrl V" for copy and paste? It says can't find element - I'm trying to target cells on an excel sheet in google sheets and they don't have unique identifiers so I need to target "by coordinates" and then submit copy and paste WITHOUT caring what element i'm "sending" to.

I was trying to automate actions at https://www.ibm.com/... (URL removed) and would click on main documentation navigation headings (on the left) but the system couldn't detect when the AJAX response finished so it could verify an element. In retrospect I think it wasn't detecting the frame I was accessing.

It can't detect bootstrap modal.

Didn't work in Google Slides: I tried to record changing color and it didn't record anything except opening Google Slides (when it was already open)

I wanted to record click/drag operations in screen space, not on page elements. UI.Vision RPA does not appear to support this.

add either the ability to send control keys or run an extension

I want mouseMove, mouseDown, mouseUp, keyboard events and timeline to record use cases for debugging purposes

Selenium IDE failed on my tests with our website (SPA, multiple frames). The error message was "can't access dead object"

compatibility with arrow, command/control etc. keys

In the website I'm testing, most links are accessed by hovering over a menu and then a submenu. When re-running a recorded macro, it failed after a time-out period when the link did not appear (because it did not perform the hover).

Probably outside the scope of the program, I'm looking for something with position based clicking rather than dom targeting.

flash support?

looked like it worked but didn't actually activate buttons within gmail app to delete emails. Nothing appeard in the Trash folder??

Add the possibility to handle the closing of the actual tab.

I needed to click on an icon extension, but it did not work

When the website is created with Angular JS, the widgets change dynamically so the IDs of the elements change and UI.Vision RPA is not able to see the elements.

doubleclick not supported,

it doesnt work on calendars year navigation,

Don't record angular webs

I need it to simulate all mouse movements not just controls being clicked or text boxes being filled

All of this can be done now - so if one if these questions is similar to yours, and you are not sure how to solve it, please ask in our forum.

Anything wrong or missing on this page? Suggestions?

...then please contact us.

Subscribe to the UI Vision RPA software newsletter . We'll send you updates on new releases that we're working on.