Consider the textual content summarization capabilities of LLMs for enhanced decision-making on AWS

Organizations throughout industries are utilizing automated textual content summarization to extra effectively deal with huge quantities of data and make higher choices. Within the monetary sector, funding banks condense earnings experiences all the way down to key takeaways to quickly analyze quarterly efficiency. Media firms use summarization to watch information and social media so journalists can rapidly write tales on creating points. Authorities companies summarize prolonged coverage paperwork and experiences to assist policymakers strategize and prioritize targets.

By creating condensed variations of lengthy, advanced paperwork, summarization know-how permits customers to give attention to probably the most salient content material. This results in higher comprehension and retention of crucial data. The time financial savings enable stakeholders to overview extra materials in much less time, gaining a broader perspective. With enhanced understanding and extra synthesized insights, organizations could make higher knowledgeable strategic choices, speed up analysis, enhance productiveness, and enhance their influence. The transformative energy of superior summarization capabilities will solely proceed rising as extra industries undertake synthetic intelligence (AI) to harness overflowing data streams.

On this submit, we discover main approaches for evaluating summarization accuracy objectively, together with ROUGE metrics, METEOR, and BERTScore. Understanding the strengths and weaknesses of those methods might help information choice and enchancment efforts. The general purpose of this submit is to demystify summarization analysis to assist groups higher benchmark efficiency on this crucial functionality as they search to maximise worth.

Varieties of summarization

Summarization can typically be divided into two most important sorts: extractive summarization and abstractive summarization. Each approaches goal to condense lengthy items of textual content into shorter varieties, capturing probably the most crucial data or essence of the unique content material, however they accomplish that in essentially other ways.

Extractive summarization entails figuring out and extracting key phrases, sentences, or segments from the unique textual content with out altering them. The system selects components of the textual content deemed most informative or consultant of the entire. Extractive summarization is beneficial if accuracy is crucial and the abstract must mirror the precise data from the unique textual content. These could possibly be use circumstances like highlighting particular authorized phrases, obligations, and rights outlined within the phrases of use. The commonest methods used for extractive summarization are time period frequency-inverse doc frequency (TF-IDF), sentence scoring, textual content rank algorithm, and supervised machine studying (ML).

Abstractive summarization goes a step additional by producing new phrases and sentences that weren’t within the authentic textual content, primarily paraphrasing and condensing the unique content material. This strategy requires a deeper understanding of the textual content, as a result of the AI must interpret the that means after which specific it in a brand new, concise kind. Massive language fashions (LLMs) are greatest fitted to abstractive summarization as a result of the transformer fashions use consideration mechanisms to give attention to related components of the enter textual content when producing summaries. The eye mechanism permits the mannequin to assign completely different weights to completely different phrases or tokens within the enter sequence, enabling it to seize long-range dependencies and contextually related data.

Along with these two major sorts, there are hybrid approaches that mix extractive and abstractive strategies. These approaches may begin with extractive summarization to establish an important content material after which use abstractive methods to rewrite or condense that content material right into a fluent abstract.

The problem

Discovering the optimum technique to guage abstract high quality stays an open problem. As organizations more and more depend on automated textual content summarization to distill key data from paperwork, the necessity grows for standardized methods to measure summarization accuracy. Ideally, these analysis metrics would quantify how nicely machine-generated summaries extract probably the most salient content material from supply texts and current coherent summaries reflecting the unique that means and context.

Nonetheless, creating strong analysis methodologies for textual content summarization presents difficulties:

• Human-authored reference summaries used for comparability usually exhibit excessive variability based mostly on subjective determinations of significance
• Nuanced facets of abstract high quality like fluency, readability, and coherence show tough to quantify programmatically
• Broad variation exists throughout summarization strategies from statistical algorithms to neural networks, complicating direct comparisons

Recall-Oriented Understudy for Gisting Analysis (ROUGE)

ROUGE metrics, equivalent to ROUGE-N and ROUGE-L, play an important function in evaluating the standard of machine-generated summaries in comparison with human-written reference summaries. These metrics give attention to assessing the overlap between the content material of machine-generated and human-crafted summaries by analyzing n-grams, that are teams of phrases or tokens. For example, ROUGE-1 evaluates the match of particular person phrases (unigrams), whereas ROUGE-2 considers pairs of phrases (bigrams). Moreover, ROUGE-N assesses the longest widespread subsequence of phrases between the 2 texts, permitting for flexibility in phrase order.

As an instance this, think about the next examples:

• ROGUE-1 metric – ROUGE-1 evaluates the overlap of unigrams (single phrases) between a generated abstract and a reference abstract. For instance, if a reference abstract incorporates “The short brown fox jumps,” and the generated abstract is “The brown fox jumps rapidly,” the ROUGE-1 metric would think about “brown,” “fox,” and “jumps” as overlapping unigrams. ROUGE-1 focuses on the presence of particular person phrases within the summaries, measuring how nicely the generated abstract captures the important thing phrases from the reference abstract.
• ROGUE-2 metric – ROUGE-2 assesses the overlap of bigrams (pairs of adjoining phrases) between a generated abstract and a reference abstract. For example, if the reference abstract has “The cat is sleeping,” and the generated abstract reads “A cat is sleeping,” ROUGE-2 would establish “cat is” and “is sleeping” as an overlapping bigram. ROUGE-2 gives perception into how nicely the generated abstract maintains the sequence and context of phrase pairs in comparison with the reference abstract.
• ROUGE-N metric – ROUGE-N is a generalized kind the place N represents any quantity, permitting analysis based mostly on n-grams (sequences of N phrases). Contemplating N=3, if the reference abstract states “The solar is shining brightly,” and the generated abstract is “Solar shining brightly,” ROUGE-3 would acknowledge “solar shining brightly” as an identical trigram. ROUGE-N presents flexibility to guage summaries based mostly on completely different lengths of phrase sequences, offering a extra complete evaluation of content material overlap.

These examples illustrate how ROUGE-1, ROUGE-2, and ROUGE-N metrics perform in evaluating automated summarization or machine translation duties by evaluating generated summaries with reference summaries based mostly on completely different ranges of phrase sequences.

Calculate a ROUGE-N rating

You should use the next steps to calculate a ROUGE-N rating:

1. Tokenize the generated abstract and the reference abstract into particular person phrases or tokens utilizing fundamental tokenization strategies like splitting by whitespace or pure language processing (NLP) libraries.
2. Generate n-grams (contiguous sequences of N phrases) from each the generated abstract and the reference abstract.
3. Depend the variety of overlapping n-grams between the generated abstract and the reference abstract.
4. Calculate precision, recall, and F1 rating:
   • Precision – The variety of overlapping n-grams divided by the whole variety of n-grams within the generated abstract.
   • Recall – The variety of overlapping n-grams divided by the whole variety of n-grams within the reference abstract.
   • F1 rating – The harmonic imply of precision and recall, calculated as (2 * precision * recall) / (precision + recall).
5. The combination F1 rating obtained from calculating precision, recall, and F1 rating for every row within the dataset is taken into account because the ROUGE-N rating.

Limitations

ROGUE has the next limitations:

• Slim give attention to lexical overlap – The core concept behind ROUGE is to check the system-generated abstract to a set of reference or human-created summaries, and measure the lexical overlap between them. This implies ROUGE has a really slim give attention to word-level similarity. It doesn’t truly consider semantic that means, coherence, or readability of the abstract. A system might obtain excessive ROUGE scores by merely extracting sentences word-for-word from the unique textual content, with out producing a coherent or concise abstract.
• Insensitivity to paraphrasing – As a result of ROUGE depends on lexical matching, it could’t detect semantic equivalence between phrases and phrases. Subsequently, paraphrasing and use of synonyms will usually result in decrease ROUGE scores, even when the that means is preserved. This disadvantages techniques that paraphrase or summarize in an abstractive means.
• Lack of semantic understanding – ROUGE doesn’t consider whether or not the system actually understood the meanings and ideas within the authentic textual content. A abstract might obtain excessive lexical overlap with references, whereas lacking the primary concepts or containing factual inconsistencies. ROUGE wouldn’t establish these points.

When to make use of ROUGE

ROUGE is straightforward and quick to calculate. Use it as a baseline or benchmark for abstract high quality associated to content material choice. ROUGE metrics are most successfully employed in situations involving abstractive summarization duties, automated summarization analysis, assessments of LLMs, and comparative analyses of various summarization approaches. Through the use of ROUGE metrics in these contexts, stakeholders can quantitatively consider the standard and effectiveness of abstract technology processes.

Metric for Analysis of Translation with Specific Ordering (METEOR)

One of many main challenges in evaluating summarization techniques is assessing how nicely the generated abstract flows logically, quite than simply choosing related phrases and phrases from the supply textual content. Merely extracting related key phrases and sentences doesn’t essentially produce a coherent and cohesive abstract. The abstract ought to circulate easily and join concepts logically, even when they aren’t introduced in the identical order as the unique doc.

The pliability of matching by lowering phrases to their root or base kind (For instance, after stemming, phrases like “operating,” “runs,” and “ran” all turn into “run”) and synonyms means METEOR correlates higher with human judgements of abstract high quality. It may establish if vital content material is preserved, even when the wording differs. This can be a key benefit over n-gram based mostly metrics like ROUGE, which solely search for precise token matches. METEOR additionally offers larger scores to summaries that concentrate on probably the most salient content material from the reference. Decrease scores are given to repetitive or irrelevant data. This aligns nicely with the purpose of summarization to maintain an important content material solely. METEOR is a semantically significant metric that may overcome a few of the limitations of n-gram matching for evaluating textual content summarization. The incorporation of stemming and synonyms permits for higher evaluation of data overlap and content material accuracy.

As an instance this, think about the next examples:

Reference Abstract: Leaves fall throughout autumn.

Generated Abstract 1: Leaves drop in fall.

Generated Abstract 2: Leaves inexperienced in summer season.

The phrases that match between the reference and generated abstract 1 are highlighted:

Reference Abstract: Leaves fall throughout autumn.

Generated Abstract 1: Leaves drop in fall.

Although “fall” and “autumn” are completely different tokens, METEOR acknowledges them as synonyms by means of its synonym matching. “Drop” and “fall” are recognized as a stemmed match. For generated abstract 2, there are not any matches with the reference abstract in addition to “Leaves,” so this abstract would obtain a a lot decrease METEOR rating. The extra semantically significant matches, the upper the METEOR rating. This enables METEOR to raised consider the content material and accuracy of summaries in comparison with easy n-gram matching.

Calculate a METEOR rating

Full the next steps to calculate a METEOR rating:

1. Tokenize the generated abstract and the reference abstract into particular person phrases or tokens utilizing fundamental tokenization strategies like splitting by whitespace or NLP libraries.
2. Calculate the unigram precision, recall, and F-mean rating, giving extra weightage to recall than precision.
3. Apply a penalty for precise matches to keep away from overemphasizing them. The penalty is chosen based mostly on dataset traits, job necessities, and the stability between precision and recall. Subtract this penalty from the F-mean rating calculated in Step 2.
4. Calculate the F-mean rating for stemmed varieties (lowering phrases to their base or root kind) and synonyms for unigrams the place relevant. Combination this with the sooner calculated F-mean rating to acquire the ultimate METEOR rating. The METEOR rating ranges from 0–1, the place 0 signifies no similarity between the generated abstract and reference abstract, and 1 signifies good alignment. Sometimes, summarization scores fall between 0–0.6.

Limitations

When using the METEOR metric for evaluating summarization duties, a number of challenges might come up:

• Semantic complexity – METEOR’s emphasis on semantic similarity can battle to seize the nuanced meanings and context in advanced summarization duties, doubtlessly resulting in inaccuracies in analysis.
• Reference variability – Variability in human-generated reference summaries can influence METEOR scores, as a result of variations in reference content material might have an effect on the analysis of machine-generated summaries.
• Linguistic variety – The effectiveness of METEOR might fluctuate throughout languages resulting from linguistic variations, syntax variations, and semantic nuances, posing challenges in multilingual summarization evaluations.
• Size discrepancy – Evaluating summaries of various lengths might be difficult for METEOR, as a result of discrepancies in size in comparison with the reference abstract might end in penalties or inaccuracies in evaluation.
• Parameter tuning – Optimizing METEOR’s parameters for various datasets and summarization duties might be time-consuming and require cautious tuning to ensure the metric gives correct evaluations.
• Analysis bias – There’s a danger of analysis bias with METEOR if not correctly adjusted or calibrated for particular summarization domains or duties. This may doubtlessly result in skewed outcomes and have an effect on the reliability of the analysis course of.

By being conscious of those challenges and contemplating them when utilizing METEOR as a metric for summarization duties, researchers and practitioners can navigate potential limitations and make extra knowledgeable choices of their analysis processes.

When to make use of METEOR

METEOR is usually used to robotically consider the standard of textual content summaries. It’s preferable to make use of METEOR as an analysis metric when the order of concepts, ideas, or entities within the abstract issues. METEOR considers the order and matches n-grams between the generated abstract and reference summaries. It rewards summaries that protect sequential data. Not like metrics like ROUGE, which depend on overlap of n-grams with reference summaries, METEOR matches stems, synonyms, and paraphrases. METEOR works higher when there might be a number of right methods of summarizing the unique textual content. METEOR incorporates WordNet synonyms and stemmed tokens when matching n-grams. Briefly, summaries which might be semantically related however use completely different phrases or phrasing will nonetheless rating nicely. METEOR has a built-in penalty for summaries with repetitive n-grams. Subsequently, it discourages word-for-word extraction or lack of abstraction. METEOR is an effective alternative when semantic similarity, order of concepts, and fluent phrasing are vital for judging abstract high quality. It’s much less acceptable for duties the place solely lexical overlap with reference summaries issues.

BERTScore

Floor-level lexical measures like ROUGE and METEOR consider summarization techniques by evaluating the phrase overlap between a candidate abstract and a reference abstract. Nonetheless, they rely closely on precise string matching between phrases and phrases. This implies they might miss semantic similarities between phrases and phrases which have completely different floor varieties however related underlying meanings. By relying solely on floor matching, these metrics might underestimate the standard of system summaries that use synonymous phrases or paraphrase ideas in a different way from reference summaries. Two summaries might convey practically similar data however obtain low surface-level scores resulting from vocabulary variations.

BERTScore is a technique to robotically consider how good a abstract is by evaluating it to a reference abstract written by a human. It makes use of BERT, a well-liked NLP approach, to know the that means and context of phrases within the candidate abstract and reference abstract. Particularly, it seems to be at every phrase or token within the candidate abstract and finds probably the most related phrase within the reference abstract based mostly on the BERT embeddings, that are vector representations of the that means and context of every phrase. It measures the similarity utilizing cosine similarity, which tells how shut the vectors are to one another. For every phrase within the candidate abstract, it finds probably the most associated phrase within the reference abstract utilizing BERT’s understanding of language. It compares all these phrase similarities throughout the entire abstract to get an general rating of how semantically related the candidate abstract is to the reference abstract. The extra related the phrases and meanings captured by BERT, the upper the BERTScore. This enables it to robotically consider the standard of a generated abstract by evaluating it to a human reference without having human analysis every time.

As an instance this, think about you will have a machine-generated abstract: “The short brown fox jumps over the lazy canine.” Now, let’s think about a human-crafted reference abstract: “A quick brown fox leaps over a sleeping canine.”

Calculate a BERTScore

Full the next steps to calculate a BERTScore:

1. BERTScore makes use of contextual embeddings to characterize every token in each the candidate (machine-generated) and reference (human-crafted) sentences. Contextual embeddings are a kind of phrase illustration in NLP that captures the that means of a phrase based mostly on its context inside a sentence or textual content. Not like conventional phrase embeddings that assign a hard and fast vector to every phrase no matter its context, contextual embeddings think about the encircling phrases to generate a novel illustration for every phrase relying on how it’s utilized in a selected sentence.
2. The metric then computes the similarity between every token within the candidate sentence with every token within the reference sentence utilizing cosine similarity. Cosine similarity helps us quantify how carefully associated two units of knowledge are by specializing in the path they level in a multi-dimensional area, making it a useful software for duties like search algorithms, NLP, and suggestion techniques.
3. By evaluating the contextual embeddings and computing similarity scores for all tokens, BERTScore generates a complete analysis that captures the semantic relevance and context of the generated abstract in comparison with the human-crafted reference.
4. The ultimate BERTScore output gives a similarity rating that displays how nicely the machine-generated abstract aligns with the reference abstract by way of that means and context.

In essence, BERTScore goes past conventional metrics by contemplating the semantic nuances and context of sentences, providing a extra subtle analysis that carefully mirrors human judgment. This superior strategy enhances the accuracy and reliability of evaluating summarization duties, making BERTScore a useful software in assessing textual content technology techniques.

Limitations:

Though BERTScore presents important benefits in evaluating summarization duties, it additionally comes with sure limitations that have to be thought-about:

• Computational depth – BERTScore might be computationally intensive resulting from its reliance on pre-trained language fashions like BERT. This may result in longer analysis occasions, particularly when processing massive volumes of textual content information.
• Dependency on pre-trained fashions – The effectiveness of BERTScore is extremely depending on the standard and relevance of the pre-trained language mannequin used. In situations the place the pre-trained mannequin might not adequately seize the nuances of the textual content, the analysis outcomes could also be affected.
• Scalability – Scaling BERTScore for giant datasets or real-time purposes might be difficult resulting from its computational calls for. Implementing BERTScore in manufacturing environments might require optimization methods to offer environment friendly efficiency.
• Area specificity – BERTScore’s efficiency might fluctuate throughout completely different domains or specialised textual content sorts. Adapting the metric to particular domains or duties might require fine-tuning or changes to supply correct evaluations.
• Interpretability – Though BERTScore gives a complete analysis based mostly on contextual embeddings, decoding the particular causes behind the similarity scores generated for every token might be advanced and will require further evaluation.
• Reference-free analysis – Though BERTScore reduces the reliance on reference summaries for analysis, this reference-free strategy might not totally seize all facets of summarization high quality, significantly in situations the place human-crafted references are important for assessing content material relevance and coherence.

Acknowledging these limitations might help you make knowledgeable choices when utilizing BERTScore as a metric for evaluating summarization duties, offering a balanced understanding of its strengths and constraints.

When to make use of BERTScore

BERTScore can consider the standard of textual content summarization by evaluating a generated abstract to a reference abstract. It makes use of neural networks like BERT to measure semantic similarity past simply precise phrase or phrase matching. This makes BERTScore very helpful when semantic constancy preserving the complete that means and content material is crucial in your summarization job. BERTScore will give larger scores to summaries that convey the identical data because the reference abstract, even when they use completely different phrases and sentence buildings. The underside line is that BERTScore is right for summarization duties the place retaining the complete semantic that means not simply key phrases or subjects is significant. Its superior neural scoring permits it to check that means past surface-level phrase matching. This makes it appropriate for circumstances the place delicate variations in wording can considerably alter general that means and implications. BERTScore, specifically, excels in capturing semantic similarity, which is essential for assessing the standard of abstractive summaries like these produced by Retrieval Augmented Era (RAG) fashions.

Mannequin analysis frameworks

Mannequin analysis frameworks are important for precisely gauging the efficiency of varied summarization fashions. These frameworks are instrumental in evaluating fashions, offering coherence between generated summaries and supply content material, and pinpointing deficiencies in analysis strategies. By conducting thorough assessments and constant benchmarking, these frameworks propel textual content summarization analysis by advocating standardized analysis practices and enabling multifaceted mannequin comparisons.

In AWS, the FMEval library inside Amazon SageMaker Clarify streamlines the analysis and number of basis fashions (FMs) for duties like textual content summarization, query answering, and classification. It empowers you to guage FMs based mostly on metrics equivalent to accuracy, robustness, creativity, bias, and toxicity, supporting each automated and human-in-the-loop evaluations for LLMs. With UI-based or programmatic evaluations, FMEval generates detailed experiences with visualizations to quantify mannequin dangers like inaccuracies, toxicity, or bias, serving to organizations align with their accountable generative AI tips. On this part, we exhibit how one can use the FMEval library.

Consider Claude v2 on summarization accuracy utilizing Amazon Bedrock

The next code snippet is an instance of how one can work together with the Anthropic Claude mannequin utilizing Python code:

import json
# We use Claude v2 on this instance.
# See https://docs.anthropic.com/claude/reference/claude-on-amazon-bedrock#list-available-models
# for directions on how one can record the mannequin IDs for all obtainable Claude mannequin variants.
model_id = 'anthropic.claude-v2'
settle for = "software/json"
contentType = "software/json"
# `prompt_data` is structured within the format that the Claude mannequin expects, as documented right here:
# https://docs.aws.amazon.com/bedrock/newest/userguide/model-parameters-claude.html#model-parameters-claude-request-body
prompt_data = """Human: Who's Barack Obama?
Assistant:
"""
# For extra particulars on parameters that may be included in `physique` (equivalent to "max_tokens_to_sample"),
# see https://docs.aws.amazon.com/bedrock/newest/userguide/model-parameters-claude.html#model-parameters-claude-request-body
physique = json.dumps({"immediate": prompt_data, "max_tokens_to_sample": 500})
# Invoke the mannequin
response = bedrock_runtime.invoke_model(
physique=physique, modelId=model_id, settle for=settle for, contentType=contentType
)
# Parse the invocation response
response_body = json.hundreds(response.get("physique").learn())
print(response_body.get("completion"))

In easy phrases, this code performs the next actions:

1. Import the required libraries, together with json, to work with JSON information.
2. Outline the mannequin ID as anthropic.claude-v2 and set the content material kind for the request.
3. Create a prompt_data variable that buildings the enter information for the Claude mannequin. On this case, it asks the query “Who’s Barack Obama?” and expects a response from the mannequin.
4. Assemble a JSON object named physique that features the immediate information, and specify further parameters like the utmost variety of tokens to generate.
5. Invoke the Claude mannequin utilizing bedrock_runtime.invoke_model with the outlined parameters.
6. Parse the response from the mannequin, extract the completion (generated textual content), and print it out.

Be sure that the AWS Identity and Access Management (IAM) function related to the Amazon SageMaker Studio consumer profile has entry to the Amazon Bedrock fashions being invoked. Discuss with Identity-based policy examples for Amazon Bedrock for steerage on greatest practices and examples of identity-based insurance policies for Amazon Bedrock.

Utilizing the FMEval library to guage the summarized output from Claude

We use the next code to guage the summarized output:

from fmeval.data_loaders.data_config import DataConfig
from fmeval.model_runners.bedrock_model_runner import BedrockModelRunner
from fmeval.constants import MIME_TYPE_JSONLINES
from fmeval.eval_algorithms.summarization_accuracy import SummarizationAccuracy
config = DataConfig(
    dataset_name="gigaword_sample",
    dataset_uri="gigaword_sample.jsonl",
    dataset_mime_type=MIME_TYPE_JSONLINES,
    model_input_location="doc",
    target_output_location="abstract"
)
bedrock_model_runner = BedrockModelRunner(
    model_id=model_id,
    output="completion",
    content_template="{"immediate": $immediate, "max_tokens_to_sample": 500}"
)
eval_algo = SummarizationAccuracy()
eval_output = eval_algo.consider(mannequin=bedrock_model_runner, dataset_config=config,
prompt_template="Human: Summarise the next textual content in a single sentence: $featurennAssistant:n", save=True)

Within the previous code snippet, to guage textual content summarization utilizing the FMEval library, we full the next steps:

1. Create a ModelRunner to carry out invocation in your LLM. The FMEval library gives built-in assist for Amazon SageMaker endpoints and Amazon SageMaker JumpStart LLMs. You too can prolong the ModelRunner interface for any LLMs hosted anyplace.
2. Use supported eval_algorithms like toxicity, summarization, accuracy, semantic, and robustness, based mostly in your analysis wants.
3. Customise the analysis configuration parameters in your particular use case.
4. Use the analysis algorithm with both built-in or customized datasets to guage your LLM mannequin. The dataset used on this case is sourced from the next GitHub repo.

Discuss with the developer guide and examples for detailed utilization of analysis algorithms.

The next desk summarizes the outcomes of the analysis.

Check out the sample notebook for more details about the summarization evaluation that we discussed in this post.

Conclusion

ROUGE, METEOR, and BERTScore all measure the quality of machine-generated summaries, but focus on different aspects like lexical overlap, fluency, or semantic similarity. Make sure to select the metric that aligns with what defines “good” for your specific summarization use case. You can also use a combination of metrics. This provides a more well-rounded evaluation and guards against potential weaknesses of any individual metric. With the right measurements, you can iteratively improve your summarizers to meet whichever notion of accuracy matters most.

Additionally, FM and LLM evaluation is necessary to be able to productionize these models at scale. With FMEval, you get a vast set of built-in algorithms across many NLP tasks, but also a scalable and flexible tool for large-scale evaluations of your own models, datasets, and algorithms. To scale up, you can use this package in your LLMOps pipelines to evaluate multiple models. To learn more about FMEval in AWS and how to use it effectively, refer to Use SageMaker Clarify to evaluate large language models. For further understanding and insights into the capabilities of SageMaker Clarify in evaluating FMs, see Amazon SageMaker Clarify Makes It Easier to Evaluate and Select Foundation Models.

About the Authors

Dinesh Kumar Subramani is a Senior Solutions Architect based in Edinburgh, Scotland. He specializes in artificial intelligence and machine learning, and is member of technical field community with in Amazon. Dinesh works closely with UK Central Government customers to solve their problems using AWS services. Outside of work, Dinesh enjoys spending quality time with his family, playing chess, and exploring a diverse range of music.

Pranav Sharma is an AWS leader driving technology and business transformation initiatives across Europe, the Middle East, and Africa. He has experience in designing and running artificial intelligence platforms in production that support millions of customers and deliver business outcomes. He has played technology and people leadership roles for Global Financial Services organizations. Outside of work, he likes to read, play tennis with his son, and watch movies.