BClick, BType & Co - Browser Vision Commands (Beta), Robotic Process Automation (RPA)

BClick (target, click type), BMove (target, mouse event), BType (text)

New in Ui.Vision Version 10

The new B command family brings real, trusted mouse and keyboard input to browser automation - without installing the Ui.Vision XModules. B commands send input events through the browser's own debugging interface (Chrome DevTools Protocol), so from the website's point of view they are indistinguishable from a real user.

They sit exactly between the classic Selenium-style Click (fast, but "synthetic" DOM events that some websites ignore) and XClick (true OS-level input, but requires the XModule and a visible browser window).

The command family

Command	What it does
BClick	Click on an image, OCR text, locator, or x,y coordinate
BClickText	Click on a text found via OCR
BClickTextRelative	Click relative to an OCR text anchor (word#R10,20)
BClickRelative	Click relative to a green/pink anchor image
BMove	Move the mouse (supports #down, #up for drag & drop)
BMoveText / BMoveTextRelative / BMoveRelative	Mouse move variants of the above
BType	Type text with real keyboard events, including ${KEY_...} tokens and key combos

Target syntax is fully XClick-compatible: you can take an existing macro and replace XClick with BClick - images ([email protected]), OCR text (ocr=Submit), Selenium locators (id=, css=, xpath=), and viewport coordinates all work the same way.

Advantages

No XModule needed. Everything runs inside the browser extension, including image search (built-in SIMD-accelerated WASM engine) and OCR (Javascript or Cloud OCR engine). Works on any OS the browser runs on - including systems where you cannot install native apps.
100% local processing. Image recognition and the default Javascript OCR run entirely on your machine - no screenshots or page data leave the browser.
Trusted, real input events. Unlike Click, which fires JavaScript DOM events, B commands dispatch input at the browser level. Websites that ignore synthetic events (canvas apps, React/Vue widgets with strict event checking, drag & drop) respond as if a human clicked.
Browser window can stay in the background. XClick moves the real OS mouse cursor, so the browser window must be visible and in front. B commands target the tab directly - you can keep working in another window while the macro runs.
Visual + OCR targeting out of the box. Find click targets by image or by text on the screen, no locators required - great for sites with unstable DOM structures.
Drag & drop and key combos. BMove with #down/#up holds mouse buttons across steps; BType supports modifier keys and ${KEY_*} tokens.
Animated cursor. During replay an animated cursor shows where the B command clicks, making debugging easy.

Limitations

Chromium only. B commands work in Chrome and Edge. Firefox does not expose the required debugger API - in Firefox, B commands stop with a clear error message telling you to use Click or XClick instead.
Browser content only. B commands cannot click outside the web page - no browser dialogs, file pickers, other apps, or the desktop. Desktop coordinates (Dx,y) and desktop vision scope are rejected with an error; use XClick/XType (XModule) for desktop automation.
Debugger banner. While a B command runs, Chrome shows its standard "UI.Vision is debugging this browser" info bar on the automated tab.
Visual search area. Image and OCR targets are searched in the visible viewport by default. Use visionLimitSearchArea with full to search the whole page (Ui.Vision scroll-captures it automatically), or element:... to restrict the search area.
OCR engines. All OCR engines work with B commands. The "Local" OCR engine (engine 99) requires the XModule to be installed - without it, use the built-in Javascript OCR (Tesseract) or the Cloud OCR engines.

Click vs. BClick vs. XClick

Aspect	Click	BClick (new)	XClick
Input method	Synthetic DOM/JavaScript event	Real browser input via debugger API (CDP)	Native OS-level input
Browsers	Chrome, Edge, Firefox	Chrome, Edge	Chrome, Edge, Firefox
Firefox support	✅ Yes	❌ No (not supported by FX yet)	✅ Yes
XModule required	No	No	Yes (free Ui.Vision XModules)
Selenium locators (id=, css=, xpath=)	✅ Yes	✅ Yes	✅ Yes
Image & OCR text targets	❌ No	✅ Yes (built-in image search + OCR)	✅ Yes (XModule image search + OCR)
Works on full page (below the fold)	✅ Yes - DOM-based, element visibility does not matter	❌ No - image/OCR search the visible viewport by default (full page via visionLimitSearchArea with full)	❌ No - image/OCR search the visible part of the screen only
Scrolls element into view	✅ Yes	✅ Yes (locator targets)	✅ Yes (locator targets)
Browser window can be in background	✅ Yes	✅ Yes	❌ No - window must be visible and in front
Seen by website as real user input	❌ No - some sites ignore it	✅ Yes	✅ Yes
Undetectable by the website (no automation traces)	❌ No	❌ No - websites can detect the attached debugger	✅ Yes - OS-level input, indistinguishable from a human
Can click outside the web page (dialogs, desktop apps)	❌ No	❌ No	✅ Yes
Speed	Fastest	Fast	Depends on screen/vision search

Which one should I use?

Click - your default for standard web forms and links. Fastest, works in every browser.
BClick - when a site ignores Click (canvas, drag & drop, "protected" widgets), when you want image/OCR targeting without installing anything, or when the macro must run while the browser is in the background.
XClick - when you need Firefox with real input, need to click browser dialogs or native apps, or already use the XModule for desktop automation.

Demo macros

The "Browser Vision (Beta)" folder (preinstalled with Version 10) contains four demo macros showing image targets, OCR text targets, relative clicks, and drag & drop with the B commands.

The ready-to-import-and-run source code of all demo macros can be found in the Open-Source RPA software Github repository.

Works in

Ui.Vision RPA for Chrome and Edge. No XModule installation required. In Firefox, B commands stop with an error message - use Click or XClick there instead.

Feedback

Feedback on the beta is very welcome in the Ui.Vision forum.

Anything wrong or missing on this page? Suggestions?

...then please contact us.

BClick, BType & Co - Browser Vision Commands (Beta)