Amazon Bedrock Brokers observability utilizing Arize AI

This submit is cowritten with John Gilhuly from Arize AI.

With Amazon Bedrock Agents, you’ll be able to construct and configure autonomous brokers in your software. An agent helps your end-users full actions primarily based on group information and consumer enter. Brokers orchestrate interactions between basis fashions (FMs), information sources, software program functions, and consumer conversations. As well as, brokers robotically name APIs to take actions and invoke information bases to complement info for these actions. By integrating brokers, you’ll be able to speed up your improvement effort to ship generative AI functions. With brokers, you’ll be able to automate duties to your prospects and reply questions for them. For instance, you’ll be able to create an agent that helps prospects course of insurance coverage claims or make journey reservations. You don’t should provision capability, handle infrastructure, or write customized code. Amazon Bedrock manages immediate engineering, reminiscence, monitoring, encryption, consumer permissions, and API invocation.

AI brokers symbolize a basic shift in how functions make choices and work together with customers. In contrast to conventional software program techniques that observe predetermined paths, AI brokers make use of advanced reasoning that always operates as a “black field.” Monitoring AI brokers presents distinctive challenges for organizations in search of to keep up reliability, effectivity, and optimum efficiency of their AI implementations.

At the moment, we’re excited to announce a brand new integration between Arize AI and Amazon Bedrock Brokers that addresses one of the crucial important challenges in AI improvement: observability. Agent observability is a vital facet of AI operations that gives deep insights into how your Amazon Bedrock brokers carry out, work together, and execute duties. It includes monitoring and analyzing hierarchical traces of agent actions, from high-level consumer requests right down to particular person API calls and power invocations. These traces kind a structured tree of occasions, serving to builders perceive the whole journey of consumer interactions by way of the agent’s decision-making course of. Key metrics that demand consideration embrace response latency, token utilization, runtime exceptions, and examine operate calling. As organizations scale their AI implementations from proof of idea to manufacturing, understanding and monitoring AI agent conduct turns into more and more crucial.

The combination between Arize AI and Amazon Bedrock Brokers supplies builders with complete observability instruments for tracing, evaluating, and monitoring AI agent functions. This answer delivers three major advantages:

• Complete traceability – Acquire visibility into each step of your agent’s execution path, from preliminary consumer question by way of information retrieval and motion execution
• Systematic analysis framework – Apply constant analysis methodologies to measure and perceive agent efficiency
• Knowledge-driven optimization – Run structured experiments to match totally different agent configurations and establish optimum settings

The Arize AI service is out there in two variations:

• Arize AX – An enterprise answer providing superior monitoring capabilities
• Arize Phoenix – An open supply service making tracing and analysis accessible to builders

On this submit, we reveal the Arize Phoenix system for tracing and analysis. Phoenix can run in your native machine, a Jupyter pocket book, a containerized deployment, or within the cloud. We discover how this integration works, its key options, and how one can implement it in your Amazon Bedrock Brokers functions to reinforce observability and keep production-grade reliability.

Answer overview

Giant language mannequin (LLM) tracing information the paths taken by requests as they propagate by way of a number of steps or elements of an LLM software. It improves the visibility of your software or system’s well being and makes it potential to debug conduct that’s troublesome to breed regionally. For instance, when a consumer interacts with an LLM software, tracing can seize the sequence of operations, akin to doc retrieval, embedding era, language mannequin invocation, and response era, to offer an in depth timeline of the request’s execution.

For an software to emit traces for evaluation, the applying have to be instrumented. Your software will be manually instrumented or be robotically instrumented. Arize Phoenix provides a set of plugins (instrumentors) you can add to your software’s startup course of that carry out automated instrumentation. These plugins accumulate traces to your software and export them (utilizing an exporter) for assortment and visualization. The Phoenix server is a collector and UI that helps you troubleshoot your software in actual time. If you run Phoenix (for instance, the px.launch_app() container), Phoenix begins receiving traces from an software that’s exporting traces to it. For Phoenix, the instrumentors are managed by way of a single repository known as OpenInference. OpenInference supplies a set of instrumentations for widespread machine studying (ML) SDKs and frameworks in quite a lot of languages. It’s a set of conventions and plugins that’s complimentary to OpenTelemetry and on-line transaction processing (OLTP) to allow tracing of AI functions. Phoenix at present helps OTLP over HTTP.

For AWS, Boto3 supplies Python bindings to AWS providers, together with Amazon Bedrock, which supplies entry to a variety of FMs. You possibly can instrument calls to those fashions utilizing OpenInference, enabling OpenTelemetry-aligned observability of functions constructed utilizing these fashions. It’s also possible to seize traces on invocations of Amazon Bedrock brokers utilizing OpenInference and examine them in Phoenix.The next high-level structure diagram reveals an LLM software created utilizing Amazon Bedrock Brokers, which has been instrumented to ship traces to the Phoenix server.

Within the following sections, we reveal how, by putting in the openinference-instrumentation-bedrock library, you’ll be able to robotically instrument interactions with Amazon Bedrock or Amazon Bedrock brokers for observability, analysis, and troubleshooting functions in Phoenix.

Stipulations

To observe this tutorial, you should have the next:

It’s also possible to clone the GitHub repo regionally to run the Jupyter notebook your self:

git clone https://github.com/awslabs/amazon-bedrock-agent-samples.git

Set up required dependencies

Start by putting in the required libraries:

%pip set up -r necessities.txt — quiet

Subsequent, import the required modules:

import time
import boto3
import logging
import os
import nest_asyncio
from phoenix.otel import register
from openinference.instrumentation import using_metadata

nest_asyncio.apply()

The arize-phoenix-otel bundle supplies a light-weight wrapper round OpenTelemetry primitives with Phoenix-aware defaults. These defaults are conscious of surroundings variables you should set to configure Phoenix within the subsequent steps, akin to:

• PHOENIX_COLLECTOR_ENDPOINT
• PHOENIX_PROJECT_NAME
• PHOENIX_CLIENT_HEADERS
• PHOENIX_API_KEY

Configure the Phoenix surroundings

Arrange the Phoenix Cloud surroundings for this tutorial. Phoenix may also be self-hosted on AWS as an alternative.

os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com“
if not os.environ.get("PHOENIX_CLIENT_HEADERS"):
os.environ["PHOENIX_CLIENT_HEADERS"] = "api_key=" + enter("Enter your Phoenix API key: ")

Join your pocket book to Phoenix with auto-instrumentation enabled:

project_name = "Amazon Bedrock Agent Instance"
tracer_provider = register(project_name=project_name, auto_instrument=True)

The auto_instrument parameter robotically locates the openinference-instrumentation-bedrock library and devices Amazon Bedrock and Amazon Bedrock Agent calls with out requiring extra configuration. Configure metadata for the span:

metadata = { "agent" : "bedrock-agent", 
            "env" : "improvement"
Metadata is used to filter search values within the dashboard
       }

Arrange an Amazon Bedrock session and agent

Earlier than utilizing Amazon Bedrock, ensure that your AWS credentials are configured appropriately. You possibly can set them up utilizing the AWS Command Line Interface (AWS CLI) or by setting surroundings variables:

session = boto3.Session()
REGION = session.region_name
bedrock_agent_runtime = session.consumer(service_name="bedrock-agent-runtime",region_name=REGION)

We assume you’ve already created an Amazon Bedrock agent. To configure the agent, use the next code:

agent_id = "XXXXXYYYYY" # ← Configure your Bedrock Agent ID
agent_alias_id = "Z0ZZZZZZ0Z" # ← Optionally set a distinct Alias ID in case you have one

Earlier than continuing to the next step, you’ll be able to validate whether or not invoke agent is working appropriately. The response will not be necessary; we’re merely testing the API name.

print(f"Attempting to invoke alias {agent_alias_id} of agent {agent_id}...")
agent_resp = bedrock_agent_runtime.invoke_agent(
    agentAliasId=agent_alias_id,
    agentId=agent_id,
    inputText="Hi there!",
    sessionId="dummy-session",
)
if "completion" in agent_resp:
    print("✅ Acquired response")
else:
    elevate ValueError(f"No 'completion' in agent response:n{agent_resp}")

Run your agent with tracing enabled

Create a operate to run your agent and seize its output:

@using_metadata(metadata)
def run(input_text):
    session_id = f"default-session1_{int(time.time())}"

    attributes = dict(
        inputText=input_text,
        agentId=agent_id,
        agentAliasId=agent_alias_id,
        sessionId=session_id,
        enableTrace=True,
    )
    response = bedrock_agent_runtime.invoke_agent(**attributes)

    # Stream the response
    for _, occasion in enumerate(response["completion"]):
        if "chunk" in occasion:
            print(occasion)
            chunk_data = occasion["chunk"]
            if "bytes" in chunk_data:
                output_text = chunk_data["bytes"].decode("utf8")
                print(output_text)
        elif "hint" in occasion:
            print(occasion["trace"])

Take a look at your agent with just a few pattern queries:

run ("What are the entire leaves for Worker 1?")
run ("If Worker 1 takes 4 trip days off, What are the entire leaves left for Worker 1?")

You must exchange these queries with the queries that your software is constructed for. After executing these instructions, it’s best to see your agent’s responses within the pocket book output. The Phoenix instrumentation is robotically capturing detailed traces of those interactions, together with information base lookups, orchestration steps, and power calls.

View captured traces in Phoenix

Navigate to your Phoenix dashboard to view the captured traces. You will notice a complete visualization of every agent invocation, together with:

• The complete dialog context
• Data base queries and outcomes
• Device or motion group calls and responses
• Agent reasoning and decision-making steps

Phoenix’s tracing and span evaluation capabilities are helpful through the prototyping and debugging levels. By instrumenting software code with Phoenix, groups acquire detailed insights into the execution circulate, making it easy to establish and resolve points. Builders can drill down into particular spans, analyze efficiency metrics, and entry related logs and metadata to streamline debugging efforts. With Phoenix’s tracing capabilities, you’ll be able to monitor the next:

• Utility latency – Determine latency bottlenecks and tackle sluggish invocations of LLMs, retrievers, and different elements inside your software, enabling you to optimize efficiency and responsiveness.
• Token utilization – Acquire an in depth breakdown of token utilization to your LLM calls, so you’ll be able to establish and optimize the costliest LLM invocations.
• Runtime exceptions – Seize and examine crucial runtime exceptions, akin to rate-limiting occasions, that may provide help to proactively tackle and mitigate potential points.
• Retrieved paperwork – Examine the paperwork retrieved throughout a retriever name, together with the rating and order by which they have been returned, to offer perception into the retrieval course of.
• Embeddings – Study the embedding textual content used for retrieval and the underlying embedding mannequin, so you’ll be able to validate and refine your embedding methods.
• LLM parameters – Examine the parameters used when calling an LLM, akin to temperature and system prompts, to facilitate optimum configuration and debugging.
• Immediate templates – Perceive the immediate templates used through the prompting step and the variables that have been utilized, so you’ll be able to fine-tune and enhance your prompting methods.
• Device descriptions – View the descriptions and performance signatures of the instruments your LLM has been given entry to, to be able to higher perceive and management your LLM’s capabilities.
• LLM operate calls – For LLMs with operate name capabilities (akin to Anthropic’s Claude, Amazon Nova, or Meta’s Llama), you’ll be able to examine the operate choice and performance messages within the enter to the LLM. This may additional provide help to debug and optimize your software.

The next screenshot reveals the Phoenix dashboard for the Amazon Bedrock agent, displaying the latency, token utilization, complete traces.

You possibly can select one of many traces to drill right down to the extent of your entire orchestration.

Consider the agent in Phoenix

Evaluating any AI software is a problem. Evaluating an agent is much more troublesome. Brokers current a novel set of analysis pitfalls to navigate. A typical analysis metric for brokers is their operate calling accuracy, in different phrases, how nicely they do at selecting the best instrument for the job. For instance, brokers can take inefficient paths and nonetheless get to the precise answer. How are you aware in the event that they took an optimum path? Moreover, unhealthy responses upstream can result in unusual responses downstream. How do you pinpoint the place an issue originated? Phoenix additionally consists of built-in LLM evaluations and code-based experiment testing. An agent is characterised by what it is aware of concerning the world, the set of actions it could carry out, and the pathway it took to get there. To judge an agent, you should consider every element. Phoenix has constructed analysis templates for each step, akin to:

You possibly can consider the person abilities and response utilizing regular LLM analysis methods, akin to retrieval analysis, classification with LLM judges, hallucination, or Q&A correctness. On this submit, we reveal analysis of agent operate calling. You should use the Agent Operate Name eval to find out how nicely a mannequin selects a instrument to make use of, extracts the precise parameters from the consumer question, and generates the instrument name code. Now that you simply’ve traced your agent within the earlier step, the following step is so as to add evaluations to measure its efficiency. A typical analysis metric for brokers is their operate calling accuracy (how nicely they do at selecting the best instrument for the job).Full the next steps:

1. Up till now, you’ve gotten simply used the lighter-weight Phoenix OTEL tracing library. To run evals, you should to put in the total library:

!pip set up -q arize-phoenix — quiet

2. Import the required analysis elements:

import re
import json
import phoenix as px
from phoenix.evals import (
TOOL_CALLING_PROMPT_RAILS_MAP,
TOOL_CALLING_PROMPT_TEMPLATE,
BedrockModel,
llm_classify,
)
from phoenix.hint import SpanEvaluations
from phoenix.hint.dsl import SpanQuery

The next is our agent operate calling immediate template:

TOOL_CALLING_PROMPT_TEMPLATE = """

You might be an analysis assistant evaluating questions and power calls to
decide whether or not the instrument known as would reply the query. The instrument
calls have been generated by a separate agent, and chosen from the checklist of
instruments supplied beneath. It's your job to determine whether or not that agent selected
the precise instrument to name.

    [BEGIN DATA]
    ************
    [Question]: {query}
    ************
    [Tool Called]: {tool_call}
    [END DATA]

Your response have to be single phrase, both "appropriate" or "incorrect",
and mustn't comprise any textual content or characters apart from that phrase.
"incorrect" implies that the chosen instrument wouldn't reply the query,
the instrument consists of info that isn't offered within the query,
or that the instrument signature consists of parameter values that do not match
the codecs specified within the instrument signatures beneath.

"appropriate" means the proper instrument name was chosen, the proper parameters
have been extracted from the query, the instrument name generated is runnable and proper,
and that no exterior info not current within the query was used
within the generated query.

    [Tool Definitions]: {tool_definitions}
"""

3. As a result of we’re solely evaluating the inputs, outputs, and performance name columns, let’s extract these right into a simpler-to-use dataframe. Phoenix supplies a way to question your span information and immediately export solely the values you care about:

question = (
SpanQuery()
.the place(
# Filter for the `LLM` span variety.
# The filter situation is a string of legitimate Python boolean expression.
"span_kind == 'LLM' and 'analysis' not in enter.worth"
)
.choose(
query="enter.worth",
outputs="output.worth",
)
)
trace_df = px.Shopper().query_spans(question, project_name=project_name)

4. The following step is to arrange these traces right into a dataframe with columns for enter, instrument name, and power definitions. Parse the JSON enter and output information to create these columns:

def extract_tool_calls(output_value):
attempt:
tool_calls = []
# Search for instrument calls inside <function_calls> tags
if "<function_calls>" in output_value:
# Discover all tool_name tags
tool_name_pattern = r"<tool_name>(.*?)</tool_name>"
tool_names = re.findall(tool_name_pattern, output_value)

# Add every discovered instrument title to the checklist
for tool_name in tool_names:
if tool_name:
tool_calls.append(tool_name)
besides Exception as e:
print(f"Error extracting instrument calls: {e}")
go

return tool_calls

5. Apply the operate to every row of trace_df.output.worth:

trace_df["tool_call"] = trace_df["outputs"].apply(
lambda x: extract_tool_calls(x) if isinstance(x, str) else []
)

# Show the instrument calls discovered
print("Device calls present in traces:", trace_df["tool_call"].sum())

6. Add instrument definitions for analysis:

trace_df["tool_definitions"] = (
"phoenix-traces retrieves the most recent hint info from Phoenix, phoenix-experiments retrieves the most recent experiment info from Phoenix, phoenix-datasets retrieves the most recent dataset info from Phoenix"
)

Now along with your dataframe ready, you need to use Phoenix’s built-in LLM-as-a-Decide template for instrument calling to judge your software. The next methodology takes within the dataframe of traces to judge, our built-in analysis immediate, the eval mannequin to make use of, and a rails object to snap responses from our mannequin to a set of binary classification responses. We additionally instruct our mannequin to offer explanations for its responses.

7. Run the instrument calling analysis:

rails = checklist(TOOL_CALLING_PROMPT_RAILS_MAP.values())

eval_model = BedrockModel(session=session, model_id="us.anthropic.claude-3-5-haiku-20241022-v1:0")

response_classifications = llm_classify(
    information=trace_df,
    template=TOOL_CALLING_PROMPT_TEMPLATE,
    mannequin=eval_model,
    rails=rails,
    provide_explanation=True,
)
response_classifications["score"] = response_classifications.apply(
    lambda x: 1 if x["label"] == "appropriate" else 0, axis=1
)

We use the next parameters:

• df – A dataframe of circumstances to judge. The dataframe should have columns to match the default template.
• query – The question made to the mannequin. In case you exported spans from Phoenix to judge, this would be the llm.input_messages column in your exported information.
• tool_call – Info on the instrument known as and parameters included. In case you exported spans from Phoenix to judge, this would be the llm.function_call column in your exported information.

1. Lastly, log the analysis outcomes to Phoenix:

px.Shopper().log_evaluations(
SpanEvaluations(eval_name="Device Calling Eval", dataframe=response_classifications),
)

After working these instructions, you will note your analysis outcomes on the Phoenix dashboard, offering insights into how successfully your agent is utilizing its obtainable instruments.

The next screenshot reveals how the instrument calling analysis attribute reveals up whenever you run the analysis.

If you develop the person hint, you’ll be able to observe that the instrument calling analysis provides a rating of 1 if the label is appropriate. Which means that agent has responded appropriately.

Conclusion

As AI brokers grow to be more and more prevalent in enterprise functions, efficient observability is essential for facilitating their reliability, efficiency, and steady enchancment. The combination of Arize AI with Amazon Bedrock Brokers supplies builders with the instruments they should construct, monitor, and improve AI agent functions with confidence. We’re excited to see how this integration will empower builders and organizations to push the boundaries of what’s potential with AI brokers.

Keep tuned for extra updates and enhancements to this integration within the coming months. To be taught extra about Amazon Bedrock Brokers and the Arize AI integration, seek advice from the Phoenix documentation and Integrating Arize AI and Amazon Bedrock Agents: A Comprehensive Guide to Tracing, Evaluation, and Monitoring.

In regards to the Authors

Ishan Singh is a Sr. Generative AI Knowledge Scientist at Amazon Net Companies, the place he helps prospects construct modern and accountable generative AI options and merchandise. With a powerful background in AI/ML, Ishan makes a speciality of constructing generative AI options that drive enterprise worth. Exterior of labor, he enjoys enjoying volleyball, exploring native bike trails, and spending time along with his spouse and canine, Beau.

John Gilhuly is the Head of Developer Relations at Arize AI, targeted on AI agent observability and analysis tooling. He holds an MBA from Stanford and a B.S. in C.S. from Duke. Previous to becoming a member of Arize, John led GTM actions at Slingshot AI, and served as a enterprise fellow at Omega Enterprise Companions. In his pre-AI life, John constructed out and ran technical go-to-market groups at Department Metrics.

Richa Gupta is a Sr. Options Architect at Amazon Net Companies. She is keen about architecting end-to-end options for purchasers. Her specialization is machine studying and the way it may be used to construct new options that result in operational excellence and drive enterprise income. Previous to becoming a member of AWS, she labored within the capability of a Software program Engineer and Options Architect, constructing options for big telecom operators. Exterior of labor, she likes to discover new locations and loves adventurous actions.

Aris Tsakpinis is a Specialist Options Architect for Generative AI, specializing in open weight fashions on Amazon Bedrock and the broader generative AI open supply panorama. Alongside his skilled position, he’s pursuing a PhD in Machine Studying Engineering on the College of Regensburg, the place his analysis focuses on utilized pure language processing in scientific domains.

Yanyan Zhang is a Senior Generative AI Knowledge Scientist at Amazon Net Companies, the place she has been engaged on cutting-edge AI/ML applied sciences as a Generative AI Specialist, serving to prospects use generative AI to realize their desired outcomes. Yanyan graduated from Texas A&M College with a PhD in Electrical Engineering. Exterior of labor, she loves touring, figuring out, and exploring new issues.

Mani Khanuja is a Principal Generative AI Specialist SA and creator of the ebook Utilized Machine Studying and Excessive-Efficiency Computing on AWS. She leads machine studying tasks in numerous domains akin to pc imaginative and prescient, pure language processing, and generative AI. She speaks at inner and exterior conferences such AWS re:Invent, Girls in Manufacturing West, YouTube webinars, and GHC 23. In her free time, she likes to go for lengthy runs alongside the seaside.

Musarath Rahamathullah is an AI/ML and GenAI Options Architect at Amazon Net Companies, specializing in media and leisure prospects. She holds a Grasp’s diploma in Analytics with a specialization in Machine Studying. She is keen about utilizing AI options within the AWS Cloud to handle buyer challenges and democratize know-how. Her skilled background features a position as a Analysis Assistant on the prestigious Indian Institute of Know-how, Chennai. Past her skilled endeavors, she is serious about inside structure, specializing in creating stunning areas to dwell.