The aiScreenXY command allows you to find the X,Y coordinates of items (icons, buttons, menu items) by describing them. The returned x,y values are parsed into the ${!ai1} and ${!ai2} variables. You can then use these variables as direct input for the XClick and XMove commands. Ui.Vision takes care of scaling the returned x,y according to your screen settings (Screen DPI). This means that the values inside ${!ai1} and ${!ai2} can differ from the raw x,y returned by Claude.
To get started, see the new ScreenXY_Browser and ScreenXY_Desktop macros or search our user forum for the tag "prompting".
The aiPrompt command is more flexible. It allows you to attach no, one or two images and then ask Claude to execute some prompt on them. You can look for items, run OCR or ask Claude to describe what is in the image. By contrast, the aiScreenXY command is optimized for finding items in screenshots. And aiScreenXY automatically parsed the LLM output and fills the ${!ai1} and ${!ai2} variables with the result
Test Computer Use and other AI commands with the demo macros.
...then please contact us.
