The aiComputerUse command integrates mouse and keyboard control of Anthropic Claude Computer Use into our Open-Source RPA software. This makes the Ui.Vision browser extension an easy way to demo and use this technology on our Mac, Linux or Windows machine.
To get started, install the Ui.Vision Chrome extension and the XModules. Then run one of the demo macros that ship with the software. For example the "CU_PlayTicTacToe" macro is fun: Sit back and watch your Ui.Vision browser extension to play a game of Tic Tac Toe all by itself. Or run CU_FillForm, and see how you can now automate form filling without specificing each field.
Ui.Vision plays TicTacToe with the help of Anthropic Computer Use.
The syntax is easy: aiComputerUse | prompt | var1. The last response of Claude is stored in the variable. See also the other AI commands.
Computer Use Demo 2: The aiComputerUse prompt is "Fill web form with artificial data".
Here are some insights we gained from using Computer Use (CU):
- CU was trained to move the mouse before clicking. This is often unncessary. Tell it "No mouse movements, only clicks". This saves time.
- Keep the screen (for desktop automation) or browser viewport (for browser automation) as small as possible. The smaller the screenshot area, the faster and cheaper the API response. Smaller screen sizes also improve the accuracy of the returned x,y values.
- Shorter prompter work often better than longer prompts. If you ask Claude for prompting help, it will write you long and often much too detailed prompts.
Prompts can be as easy as "Fill out this form with random data" or "Search for a flight". To get started, see the new Computer Use demo macros for example prompts that work. Or search our user forum for the tag "prompting".
- In general, we find that - at least for now - Computer Use works better for browser automation than desktop automation. Google's Project Jarvis team seems to have realized this as well.
The aiScreenXY command is optimized for finding items in screenshots. Compared to aiComputerUse, this is a very narrow task. But unlike aiComputerUse the output of aiScreenXY is very predictable. Plus, aiScreenXY autotomatically stores x,y values in the ${!ai1} and ${!ai2} variables for further use with XClick and XMove.
Test Computer Use and other AI commands with the demo macros.
...then please contact us.
