Anthropic just announced an exciting new capability for their AI assistant: Claude computer use, which allows it to interact with your computer by looking at your screen and controlling your mouse and keyboard.

How to use Claude computer use

Using Claude computer use is surprisingly straightforward: the system works by letting Claude see your screen through screenshots and then giving you instructions about what to do next.

Getting started is made easy with the Claude computer use demo, which Anthropic has provided on GitHub. While there isn’t yet an official Claude computer use tutorial, the Docker demo environment is helpful. The Claude computer use Docker setup is the quickest way to begin.

We tested the Anthropic Claude computer use features with tasks like opening a web browser, installing software, and editing computer code. So how good is it?

Computer use security

What makes this particularly impressive is Claude’s ability to “precisely” (kind of, see our test video below) identify where to click on the screen. Previous AI models struggled even with giving a ballpark estimate of x/y coordinates of items in a screenshot.

However, Anthropic is being appropriately cautious about security risks, particularly warning about potential misuse through malicious websites that might try to override Claude’s normal behavior. Mouse clicks themselves are not the key risk here. The main risk is - according to our test - the fact that Claude can also run shell commands. Shell commands can do more harm more quickly than most mouse clicks. Especially because you can monitor Claude’s mouse clicks visually, but the shell commands are executed in the background.

Computer use test

So what did we test and how did Claude Computer Use work:

• Open Browser (ok)

• Install Browser extension (ok)

• Add extension icon to browser toolbar (failed, wrong approach)

• Select macro and run it (ok)

• Code edit (failed: Knew what to do, but fat fingered the edit)

• Notepad edit test (ok, fat fingered edit but found workaround)

The video below shows the test and their results:

Just as a ballpark number for the Computer use pricing: Doing the above test sequence cost us around two US$ in Anthropic API usage credits.

Summary

Anthropic Computer use is the first meaningful progress in the RPA software space in years.

We would not recommend Anthropic Computer use for any kind of production use (yet). Computer use is still slow and error prone. But the use of LLM (Large Language Models) finally moves the needle to true RPA (Robotic Process Automation).

For years, we at the UI.Vision open-source RPA software team argued that visual computer automation is the future. It is great that we found a strong (informal) ally with Claude and Anthropic.