Getting began with pc use in Amazon Bedrock Brokers

Computer use is a breakthrough functionality from Anthropic that permits basis fashions (FMs) to visually understand and interpret digital interfaces. This functionality permits Anthropic’s Claude fashions to establish what’s on a display screen, perceive the context of UI components, and acknowledge actions that ought to be carried out equivalent to clicking buttons, typing textual content, scrolling, and navigating between purposes. Nonetheless, the mannequin itself doesn’t execute these actions—it requires an orchestration layer to soundly implement the supported actions.

Right this moment, we’re saying computer use help inside Amazon Bedrock Agents utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude Sonnet 3.7 fashions on Amazon Bedrock. This integration brings Anthropic’s visible notion capabilities as a managed device inside Amazon Bedrock Brokers, offering you with a safe, traceable, and managed technique to implement pc use automation in your workflows.

Organizations throughout industries wrestle with automating repetitive duties that span a number of purposes and techniques of document. Whether or not processing invoices, updating buyer information, or managing human useful resource (HR) paperwork, these workflows typically require staff to manually switch data between completely different techniques – a course of that’s time-consuming, error-prone, and tough to scale.

Conventional automation approaches require customized API integrations for every utility, creating important improvement overhead. Pc use capabilities change this paradigm by permitting machines to understand present interfaces simply as people.

On this put up, we create a pc use agent demo that gives the important orchestration layer that transforms pc use from a notion functionality into actionable automation. With out this orchestration layer, pc use would solely establish potential actions with out executing them. The pc use agent demo powered by Amazon Bedrock Brokers offers the next advantages:

• Safe execution atmosphere – Execution of pc use instruments in a sandbox atmosphere with restricted entry to the AWS ecosystem and the net. It’s essential to notice that at present Amazon Bedrock Agent doesn’t present a sandbox atmosphere
• Complete logging – Capacity to trace every motion and interplay for auditing and debugging
• Detailed tracing capabilities – Visibility into every step of the automated workflow
• Simplified testing and experimentation – Diminished danger when working with this experimental functionality by way of managed controls
• Seamless orchestration – Coordination of advanced workflows throughout a number of techniques with out customized code

This integration combines Anthropic’s perceptual understanding of digital interfaces with the orchestration capabilities of Amazon Bedrock Brokers, creating a strong agent for automating advanced workflows throughout purposes. Reasonably than construct customized integrations for every system, builders can now create brokers that understand and work together with present interfaces in a managed, safe means.

With pc use, Amazon Bedrock Brokers can automate duties by way of fundamental GUI actions and built-in Linux instructions. For instance, your agent might take screenshots, create and edit textual content recordsdata, and run built-in Linux instructions. Utilizing Amazon Bedrock Brokers and appropriate Anthropic’s Claude fashions, you should use the next motion teams:

• Pc device – Allows interactions with consumer interfaces (clicking, typing, scrolling)
• Textual content editor device – Supplies capabilities to edit and manipulate recordsdata
• Bash – Permits execution of built-in Linux instructions

Answer overview

An instance pc use workflow consists of the next steps:

1. Create an Amazon Bedrock agent and use pure language to explain what the agent ought to do and the way it ought to work together with customers, for instance: “You might be pc use agent able to utilizing Firefox internet browser for internet search.”
2. Add the Amazon Bedrock Brokers supported pc use motion teams to your agent utilizing CreateAgentActionGroup API.
3. Invoke the agent with a consumer question that requires pc use instruments, for instance, “What’s Amazon Bedrock, are you able to search the net?”
4. The Amazon Bedrock agent makes use of the device definitions at its disposal and decides to make use of the pc motion group to click on a screenshot of the atmosphere. Utilizing the return control functionality of Amazon Bedrock Brokers, the agent the responds with the device or instruments that it needs to execute. The return management functionality is required for utilizing pc use with Amazon Bedrock Brokers.
5. The workflow parses the agent response and executes the device returned in a sandbox atmosphere. The output is given again to the Amazon Bedrock agent for additional processing.
6. The Amazon Bedrock agent continues to reply with instruments at its disposal till the duty is full.

You may recreate this instance within the us-west-2 AWS Area with the AWS Cloud Development Kit (AWS CDK) by following the directions within the GitHub repository. This demo deploys a containerized utility utilizing AWS Fargate throughout two Availability Zones within the us-west-2 Area. The infrastructure operates inside a digital personal cloud (VPC) containing public subnets in every Availability Zone, with an web gateway offering exterior connectivity. The structure is complemented by important supporting providers, together with AWS Key Management Service (AWS KMS) for safety and Amazon CloudWatch for monitoring, making a resilient, serverless container atmosphere that alleviates the necessity to handle underlying infrastructure whereas sustaining sturdy safety and excessive availability.

The next diagram illustrates the answer structure.

On the core of our answer are two Fargate containers managed by way of Amazon Elastic Container Service (Amazon ECS), every protected by its personal safety group. The primary is our orchestration container, which not solely handles the communication between Amazon Bedrock Brokers and finish customers, but in addition orchestrates the workflow that allows device execution. The second is the environment container, which serves as a safe sandbox the place the Amazon Bedrock agent can safely run its pc use instruments. The atmosphere container has restricted entry to the remainder of the ecosystem and the web. We make the most of service discovery to attach Amazon ECS providers with DNS names.

The orchestration container contains the next elements:

• Streamlit UI – The Streamlit UI that facilitates interplay between the tip consumer and pc use agent
• Return management loop – The workflow liable for parsing the instruments that the agent needs to execute and returning the output of those instruments

The atmosphere container contains the next elements:

• UI and pre-installed purposes – A light-weight UI and pre-installed Linux purposes like Firefox that can be utilized to finish the consumer’s duties
• Instrument implementation – Code that may execute pc use device within the atmosphere like “screenshot” or “double-click”
• Quart (RESTful) JSON API – An orchestration container that makes use of Quart to execute instruments in a sandbox atmosphere

The next diagram illustrates these elements.

Stipulations

1. AWS Command Line Interface (CLI), comply with directions here. Be sure that to setup credentials, comply with directions here.
2. Require Python 3.11 or later.
3. Require Node.js 14.15.0 or later.
4. AWS CDK CLI, comply with directions here.
5. Allow model access for Anthropic’s Claude Sonnet 3.5 V2 and for Anthropic’s Claude Sonnet 3.7.
6. Boto3 model >= 1.37.10.

Create an Amazon Bedrock agent with pc use

You should utilize the next code pattern to create a easy Amazon Bedrock agent with pc, bash, and textual content editor motion teams. It’s essential to supply a appropriate motion group signature when utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude 3.7 Sonnet as highlighted here.

import boto3
import time

# Step 1: Create the bedrock agent shopper

bedrock_agent = boto3.shopper("bedrock-agent", region_name="us-west-2")

# Step 2: Create an agent

create_agent_response = create_agent_response = bedrock_agent.create_agent(
        agentResourceRoleArn=agent_role_arn, # Amazon Bedrock Agent execution position
        agentName="computeruse",
        description="""Instance agent for pc use. 
				This agent ought to solely function on 
				Sandbox environments with restricted privileges.""",
        foundationModel="us.anthropic.claude-3-7-sonnet-20250219-v1:0",      
		instruction="""You might be pc use agent able to utilizing Firefox 
                 internet browser for internet search.""",
)

time.sleep(30) # await agent to be created

# Step 3.1: Create and fix pc motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="ComputerActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Pc",
    parentActionGroupSignatureParams={
        "sort": "computer_20250124",
        "display_height_px": "768",
        "display_width_px": "1024",
        "display_number": "1",
    },
)

# Step 3.2: Create and fix bash motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="BashActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Bash",
    parentActionGroupSignatureParams={
        "sort": "bash_20250124",
    },
)

# Step 3.3: Create and fix textual content editor motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="TextEditorActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.TextEditor",
    parentActionGroupSignatureParams={
        "sort": "text_editor_20250124",
    },
)

# Step 3.4 Create Climate Motion Group

bedrock_agent.create_agent_action_group(
        actionGroupName="WeatherActionGroup",
        agentId=create_agent_response["agent"]["agentId"],
        agentVersion="DRAFT",
        actionGroupExecutor = {
            'customControl': 'RETURN_CONTROL',
        },
        functionSchema = {
            'capabilities': [
                {
                    "name": "get_current_weather",
                    "description": "Get the current weather in a given location.",
                    "parameters": {
                        "location": {
                            "type": "string",
                            "description": "The city, e.g., San Francisco",
                            "required": True,
                        },
                        "unit": {
                            "type": "string",
                            "description": 'The unit to use, e.g., 
									fahrenheit or celsius. Defaults to "fahrenheit"',
                            "required": False,
                        },
                    },
                    "requireConfirmation": "DISABLED",
                }
            ]
        },
)
time.sleep(10)
# Step 4: Put together agent

bedrock_agent.prepare_agent(agentId=create_agent_response["agent"]["agentId"])

Instance use case

On this put up, we display an instance the place we use Amazon Bedrock Brokers with the pc use functionality to finish an online kind. Within the instance, the pc use agent can even swap Firefox tabs to work together with a customer relationship management (CRM) agent to get the required data to finish the shape. Though this instance makes use of a pattern CRM utility because the system of document, the identical method works with Salesforce, SAP, Workday, or different techniques of document with the suitable authentication frameworks in place.

Within the demonstrated use case, you possibly can observe how nicely the Amazon Bedrock agent carried out with pc use instruments. Our implementation accomplished the client ID, buyer identify, and e-mail by visually inspecting the excel information. Nonetheless, for the overview, it determined to pick the cell and replica the information, as a result of the data wasn’t utterly seen on the display screen. Lastly, the CRM agent was used to get extra data on the client.

Greatest practices

The next are some methods you possibly can enhance the efficiency on your use case:

Concerns

The pc use characteristic is made accessible to you as a beta service as outlined within the AWS Service Phrases. It’s topic to your settlement with AWS and the AWS Service Phrases, and the relevant mannequin EULA. Pc use poses distinctive dangers which might be distinct from normal API options or chat interfaces. These dangers are heightened when utilizing the pc use characteristic to work together with the web. To attenuate dangers, contemplate taking precautions equivalent to:

• Function pc use performance in a devoted digital machine or container with minimal privileges to attenuate direct system exploits or accidents
• To assist stop data theft, keep away from giving the pc use API entry to delicate accounts or information
• Restrict the pc use API’s web entry to required domains to scale back publicity to malicious content material
• To implement correct oversight, preserve a human within the loop for delicate duties (equivalent to making selections that would have significant real-world penalties) and for something requiring affirmative consent (equivalent to accepting cookies, executing monetary transactions, or agreeing to phrases of service)

Any content material that you just allow Anthropic’s Claude to see or entry can probably override directions or trigger the mannequin to make errors or carry out unintended actions. Taking correct precautions, equivalent to isolating Anthropic’s Claude from delicate surfaces, is important – together with to keep away from dangers associated to immediate injection. Earlier than enabling or requesting permissions essential to allow pc use options in your individual merchandise, inform finish customers of any related dangers, and acquire their consent as acceptable.

Clear up

If you find yourself performed utilizing this answer, ensure to wash up all of the assets. Observe the directions within the offered GitHub repository.

Conclusion

Organizations throughout industries face important challenges with cross-application workflows that historically require handbook information entry or advanced customized integrations. The mixing of Anthropic’s pc use functionality with Amazon Bedrock Brokers represents a transformative method to those challenges.

Through the use of Amazon Bedrock Brokers because the orchestration layer, organizations can alleviate the necessity for customized API improvement for every utility, profit from complete logging and tracing capabilities important for enterprise deployment, and implement automation options shortly.

As you start exploring pc use with Amazon Bedrock Brokers, contemplate workflows in your group that would profit from this method. From bill processing to buyer onboarding, HR documentation to compliance reporting, the potential purposes are huge and transformative.

We’re excited to see how you’ll use Amazon Bedrock Brokers with the pc use functionality to securely streamline operations and reimagine enterprise processes by way of AI-driven automation.

Assets

To be taught extra, consult with the next assets:

Concerning the Authors

Eashan Kaushik is a Specialist Options Architect AI/ML at Amazon Net Companies. He’s pushed by creating cutting-edge generative AI options whereas prioritizing a customer-centric method to his work. Earlier than this position, he obtained an MS in Pc Science from NYU Tandon College of Engineering. Exterior of labor, he enjoys sports activities, lifting, and working marathons.

Maira Ladeira Tanke is a Tech Lead for Agentic workloads in Amazon Bedrock at AWS, the place she permits prospects on their journey to develop autonomous AI techniques. With over 10 years of expertise in AI/ML. At AWS, Maira companions with enterprise prospects to speed up the adoption of agentic purposes utilizing Amazon Bedrock, serving to organizations harness the ability of basis fashions to drive innovation and enterprise transformation. In her free time, Maira enjoys touring, taking part in along with her cat, and spending time along with her household someplace heat.

Raj Pathak is a Principal Options Architect and Technical advisor to Fortune 50 and Mid-Sized FSI (Banking, Insurance coverage, Capital Markets) prospects throughout Canada and the US. Raj focuses on Machine Studying with purposes in Generative AI, Pure Language Processing, Clever Doc Processing, and MLOps.

Adarsh Srikanth is a Software program Growth Engineer at Amazon Bedrock, the place he develops AI agent providers. He holds a grasp’s diploma in pc science from USC and brings three years of trade expertise to his position. He spends his free time exploring nationwide parks, discovering new mountaineering trails, and taking part in numerous racquet sports activities.

Abishek Kumar is a Senior Software program Engineer at Amazon, bringing over 6 years of invaluable expertise throughout each retail and AWS organizations. He has demonstrated experience in creating generative AI and machine studying options, particularly contributing to key AWS providers together with SageMaker Autopilot, SageMaker Canvas, and AWS Bedrock Brokers. All through his profession, Abishek has proven ardour for fixing advanced issues and architecting large-scale techniques that serve thousands and thousands of shoppers worldwide. When not immersed in expertise, he enjoys exploring nature by way of mountaineering and touring adventures along with his spouse.

Krishna Gourishetti is a Senior Software program Engineer for the Bedrock Brokers workforce in AWS. He’s captivated with constructing scalable software program options that resolve buyer issues. In his free time, Krishna likes to go on hikes.