Simplify multimodal generative AI with Amazon Bedrock Knowledge Automation

Builders face vital challenges when utilizing foundation models (FMs) to extract knowledge from unstructured belongings. This knowledge extraction course of requires rigorously figuring out fashions that meet the developer’s particular accuracy, price, and have necessities. Moreover, builders should make investments appreciable time optimizing worth efficiency by fine-tuning and in depth immediate engineering. Managing a number of fashions, implementing security guardrails, and adapting outputs to align with downstream system necessities could be tough and time consuming.

Amazon Bedrock Data Automation in public preview helps handle these and different challenges. This new functionality from Amazon Bedrock affords a unified expertise for builders of all skillsets to simply automate the extraction, transformation, and technology of related insights from paperwork, photos, audio, and movies to construct generative AI–powered functions. With Amazon Bedrock Knowledge Automation, prospects can totally make the most of their knowledge by extracting insights from their unstructured multimodal content material in a format appropriate with their functions. Amazon Bedrock Knowledge Automation’s managed expertise, ease of use, and customization capabilities assist prospects ship enterprise worth sooner, eliminating the necessity to spend effort and time orchestrating a number of fashions, engineering prompts, or stitching collectively outputs.

On this submit, we display the best way to use Amazon Bedrock Knowledge Automation within the AWS Management Console and the AWS SDK for Python (Boto3) for media evaluation and clever doc processing (IDP) workflows.

Amazon Bedrock Knowledge Automation overview

You should use Amazon Bedrock Knowledge Automation to generate customary outputs and customized outputs. Normal outputs are modality-specific default insights, akin to video summaries that seize key moments, visible and audible poisonous content material, explanations of doc charts, graph determine knowledge, and extra. Customized outputs use customer-defined blueprints that specify output necessities utilizing pure language or a schema editor. The blueprint features a record of fields to extract, knowledge format for every area, and different directions, akin to knowledge transformations and normalizations. This offers prospects full management of the output, making it straightforward to combine Amazon Bedrock Knowledge Automation into present functions.

Utilizing Amazon Bedrock Knowledge Automation, you possibly can construct highly effective generative AI functions and automate use instances akin to media evaluation and IDP. Amazon Bedrock Knowledge Automation can also be built-in with Amazon Bedrock Knowledge Bases, making it simpler for builders to generate significant data from their unstructured multimodal content material to offer extra related responses for Retrieval Augmented Generation (RAG).

Prospects can get began with customary outputs for all 4 modalities: paperwork, photos, movies, and audio and customized outputs for paperwork and pictures. Customized outputs for video and audio will probably be supported when the potential is mostly accessible.

Amazon Bedrock Knowledge Automation for photos, audio, and video

To take a media evaluation instance, suppose that prospects within the media and leisure business want to monetize long-form content material, akin to TV exhibits and films, by contextual advert placement. To ship the precise advertisements on the proper video moments, that you must derive significant insights from each the advertisements and the video content material. Amazon Bedrock Knowledge Automation permits your contextual advert placement software by producing these insights. For example, you possibly can extract priceless data akin to video summaries, scene-level summaries, content material moderation ideas, and scene classifications based mostly on the Interactive Promoting Bureau (IAB) taxonomy.

To get began with deriving insights with Amazon Bedrock Knowledge Automation, you possibly can create a mission the place you possibly can specify your output configuration utilizing the AWS console, AWS Command Line Interface (AWS CLI) or API.

To create a mission on the Amazon Bedrock console, observe these steps:

1. Develop the Knowledge Automation dropdown menu within the navigation pane and choose Tasks, as proven within the following screenshot.
   
2. From the Tasks console, create a brand new mission and supply a mission identify, as proven within the following screenshot.
   
3. From inside the mission, select Edit, as proven within the following screenshot, to specify or modify an output configuration. Normal output is the default means of interacting with Amazon Bedrock Knowledge Automation, and it may be used with audio, paperwork, photos and movies, the place you possibly can have one customary output configuration per knowledge kind for every mission.
   
4. For purchasers who need to analyze photos and movies for media evaluation, customary output can be utilized to generate insights akin to picture abstract, video scene abstract, and scene classifications with IAB taxonomy. You may choose the picture summarization, video scene summarization, and IAB taxonomy checkboxes from the Normal output tab after which select Save adjustments to complete configuring your mission, as proven within the following screenshot.
   
5. To check the usual output configuration utilizing your media belongings, select Take a look at, as proven within the following screenshot.
   

The subsequent instance makes use of the mission to generate insights for a journey advert.

1. Add a picture, then select Generate outcomes, as proven within the following screenshot, for Amazon Bedrock Knowledge Automation to invoke an inference request.
   
2. Amazon Bedrock Knowledge Automation will course of the uploaded file based mostly on the mission’s configuration, robotically detecting that the file is a picture after which producing a abstract and IAB classes for the journey advert.
   
3. After you’ve got generated insights for the advert picture, you possibly can generate video insights to find out the very best video scene for efficient advert placement. In the identical mission, add a video file and select Generate outcomes, as proven within the following screenshot.
   

Amazon Bedrock Knowledge Automation will detect that the file is a video and can generate insights for the video based mostly on the usual output configuration specified within the mission, as proven within the following screenshot.

These insights from Amazon Bedrock Knowledge Automation, might help you successfully place related advertisements in your video content material, which might help enhance content material monetization.

Clever doc processing with Amazon Bedrock Knowledge Automation

You should use Amazon Bedrock Knowledge Automation to automate IDP workflows at scale, without having to orchestrate complicated doc processing duties akin to classification, extraction, normalization, or validation.

To take a mortgage instance, a lender desires to automate the processing of a mortgage lending packet to streamline their IDP pipeline and enhance the accuracy of mortgage processing. Amazon Bedrock Knowledge Automation simplifies the automation of complicated IDP duties akin to doc splitting, classification, knowledge extraction, output format normalization, and knowledge validation. Amazon Bedrock Knowledge Automation additionally incorporates confidence scores and visible grounding of the output knowledge to mitigate hallucinations and assist enhance end result reliability.

For instance, you possibly can generate customized output by defining blueprints, which specify output necessities utilizing pure language or a schema editor, to course of a number of file sorts in a single, streamlined API. Blueprints could be created utilizing the console or the API, and you should utilize a catalog blueprint or create a customized blueprint for paperwork and pictures.

For all modalities, this workflow consists of three important steps: making a mission, invoking the evaluation, and retrieving the outcomes.

The next resolution walks you thru a simplified mortgage lending course of with Amazon Bedrock Knowledge Automation utilizing the Amazon SDK for Python (Boto3), which is easy to combine into an present IDP workflow.

Stipulations

Earlier than you invoke the Amazon Bedrock API, be sure you have the next:

Create customized blueprint

On this instance, you’ve got the lending packet, as proven within the following picture, which comprises three paperwork: a pay stub, a W-2 kind, and a driver’s license.

Amazon Bedrock Knowledge Automation has pattern blueprints for these three paperwork that outline generally extracted fields. Nevertheless, you may also customise Amazon Bedrock Knowledge Automation to extract particular fields from every doc. For instance, you possibly can extract solely the gross pay and web pay from the pay stub by making a customized blueprint.

To create a customized blueprint utilizing the API, you should utilize the CreateBlueprint operation utilizing the Amazon Bedrock Data Automation Client. The next instance exhibits the gross pay and web pay being outlined as properties handed to CreateBlueprint, to be extracted from the lending packet:

bda_create_blueprint_response = bedrock_data_automation_client.create_blueprint(
    blueprintName="CUSTOM_PAYSLIP_BLUEPRINT",
    kind="DOCUMENT",
    blueprintStage="LIVE",
    schema=json.dumps({
        '$schema': 'http://json-schema.org/draft-07/schema#',
        'description': 'default',
        'documentClass': 'default',
        'kind': 'object',
        'properties': {
            'gross_pay_this_period': {
                'kind': 'quantity',
                'inferenceType': 'extractive',
                'description': 'The gross pay for this pay interval from the Earnings desk'
            },
            'net_pay': {
                'kind': 'quantity',
                'inferenceType': 'extractive',
                'description': 'The web pay for this pay interval from the underside of the doc'
            }
        }
    }),
)

The CreateBlueprint response returns the blueprintARN for the pay stub’s customized blueprint:

'blueprintArn: arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:blueprint/<BLUEPRINT_ID>'

Configure Amazon Bedrock Knowledge Automation mission

To start processing recordsdata utilizing blueprints with Amazon Bedrock Knowledge Automation, you first have to create a knowledge automation mission. To course of a multiple-page doc containing completely different file sorts, you possibly can configure a mission with completely different blueprints for every file kind.

Use Amazon Bedrock Knowledge Automation to use a number of doc blueprints inside one mission so you possibly can course of several types of paperwork inside the similar mission, every with its personal customized extraction logic.

When utilizing the API to create a mission, you invoke the CreateDataAutomationProject operation. The next is an instance of how one can configure customized output utilizing the customized blueprint for the pay stub and the pattern blueprints for the W-2 and driver’s license:

bda_bedrock_automation_create_project_response = bedrock_data_automation_client.create_data_automation_project(
    projectName="TEST_PROJECT",
    projectDescription='take a look at BDA mission',
    projectStage=bda_stage,
    standardOutputConfiguration={
        'doc': {
            'outputFormat': {
                'textFormat': {
                    'sorts': ['PLAIN_TEXT']
                },
                'additionalFileFormat': {
                    'state': 'ENABLED',
                }
            }
        },
    },
    customOutputConfiguration={
        'blueprints': [
          {
              'blueprintArn': 'arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:blueprint/<BLUEPRINT_ID>'
          },
          {
              'blueprintArn': 'arn:aws:bedrock:us-west-2:aws:blueprint/bedrock-data-automation-public-w2-form'
          },
          {
              'blueprintArn': 'arn:aws:bedrock:us-west-2:aws:blueprint/bedrock-data-automation-public-us-driver-license'
          },
        ],
    },
    overrideConfiguration={
        'doc': {
            'splitter': {
                'state': 'ENABLED'
            }
        }
    },
)

The CreateProject response returns the projectARN for the mission:

'arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:data-automation-project/<PROJECT_ID>'

To course of several types of paperwork utilizing a number of doc blueprints in a single mission, Amazon Bedrock Knowledge Automation makes use of a splitter configuration, which should be enabled by the API. The next is the override configuration for the splitter, and you’ll check with the Boto3 documentation for extra data:

overrideConfiguration={
    'doc': {
        'splitter':  'DISABLED'
        
    }
},

Upon creation, the API validates the enter configuration and creates a brand new mission, returning the projectARN, as proven within the following screenshot.

'arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:data-automation-project/<PROJECT_ID>'

Take a look at the answer

Now that the blueprint and mission setup is full, the InvokeDataAutomationAsync operation from the Amazon Bedrock Data Automation runtime can be utilized to start out processing recordsdata. This API name initiatives the asynchronous processing of recordsdata in an S3 bucket, on this case the lending packet, utilizing the configuration outlined within the mission by passing the mission’s ARN:

bda_invoke_data_automation_async_response = bedrock_data_automation_runtime_client.invoke_data_automation_async(
    inputConfiguration={'s3Uri': '<S3_URI>'},
    outputConfiguration={'s3Uri': '<S3_URI>'},
    dataAutomationConfiguration={
        'dataAutomationArn': 'arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:data-automation-project/<PROJECT_ID>',
        'stage': 'LIVE'
    }
)

InvokeDataAutomationAsync returns the invocationARN:

'arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:data-automation-invocation/<INVOCATION_ID>'

GetDataAutomationStatus can be utilized to view the standing of the invocation, utilizing the InvocationARN from the earlier response:

bda_invoke_data_automation_async_response = bedrock_data_automation_runtime_client.get_data_automation_status(
    invocationArn='arn:aws:bedrock:us-west-2:<AWS_ACCOUNT_ID>:data-automation-invocation/<INVOCATION_ID>'
)

When the job is full, view the leads to the S3 bucket used within the outputConfiguration by navigating to the ~/JOB_ID/0/custom_output/ folder.

From the next pattern output, Amazon Bedrock Knowledge Automation related the pay stub file with the customized pay stub blueprint with a excessive degree of confidence:

'matched_blueprint': {
    'arn': '<BLUEPRINT_ARN>', 'identify': 'CUSTOM_PAYSLIP_BLUEPRINT', 'confidence': 0.99959725
}

Utilizing the matched blueprint, Amazon Bedrock Knowledge Automation was in a position to precisely extract every area outlined within the blueprint:

'inference_result': {
    'net_pay': 291.9, 'gross_pay_this_period': 452.43
}

Moreover, Amazon Bedrock Knowledge Automation returns confidence intervals and bounding field data for every area:

'explainability_info': [{
    'net_pay': {'success': true, 'confidence': 0.96484375, 'geometry': [{'boundingBox': ...

This example demonstrates how customers can use Amazon Bedrock Data Automation to streamline and automate an IDP workflow. Amazon Bedrock Data Automation automates complex document processing tasks such as data extraction, normalization, and validation from documents. Amazon Bedrock Data Automation helps to reduce operational complexity and improves processing efficiency to handle higher loan processing volumes, minimize errors, and drive operational excellence.

Cleanup

When you’re finished evaluating this feature, delete the S3 bucket and any objects to avoid any further charges.

Summary

Customers can get started with Amazon Bedrock Data Automation, which is available in public preview in AWS Region US West 2 (Oregon). Learn more on Amazon Bedrock Data Automation and how to automate the generation of accurate information from unstructured content for building generative AI–based applications.

About the authors

Ian Lodge is a Solutions Architect at AWS, helping ISV customers in solving their architectural, operational, and cost optimization challenges. Outside of work he enjoys spending time with his family, ice hockey and woodworking.

Alex Pieri is a Solutions Architect at AWS that works with retail customers to plan, build, and optimize their AWS cloud environments. He specializes in helping customers build enterprise-ready generative AI solutions on AWS.

Raj Pathak is a Principal Solutions Architect and Technical advisor to Fortune 50 and Mid-Sized FSI (Banking, Insurance, Capital Markets) customers across Canada and the United States. Raj specializes in Machine Learning with applications in Generative AI, Natural Language Processing, Intelligent Document Processing, and MLOps.