Make movies accessible with automated audio descriptions utilizing Amazon Nova

In accordance with the World Well being Group, more than 2.2 billion people globally have vision impairment. For compliance with incapacity laws, such because the Americans with Disabilities Act (ADA) in the USA, media in visible codecs like tv exhibits or films are required to offer accessibility to visually impaired folks. This typically comes within the type of audio description tracks that narrate the visible parts of the movie or present. In accordance with the International Documentary Association, creating audio descriptions can price $25 per minute (or extra) when utilizing third events. For constructing audio descriptions internally, the hassle for companies within the media trade may be important, requiring content creators, audio description writers, description narrators, audio engineers, delivery vendors and extra in line with the American Council of the Blind (ACB). This results in the pure query, are you able to automate this course of with the assistance of generative AI choices in Amazon Web Services (AWS)?

Newly introduced in December at re:Invent 2024, the Amazon Nova Foundation Models household is out there by Amazon Bedrock and contains three multimodal foundational models (FMs):

• Amazon Nova Lite (GA) – A low-cost multimodal mannequin that’s lightning-fast for processing picture, video, and textual content inputs
• Amazon Nova Professional (GA) – A extremely succesful multimodal mannequin with a balanced mixture of accuracy, pace, and value for a variety of duties
• Amazon Nova Premier (GA) – Our most succesful mannequin for complicated duties and a trainer for mannequin distillation

On this publish, we show how you should utilize companies like Amazon Nova, Amazon Rekognition, and Amazon Polly to automate the creation of accessible audio descriptions for video content material. This strategy can considerably cut back the time and value required to make movies accessible for visually impaired audiences. Nonetheless, this publish doesn’t present a whole, deployment-ready resolution. We share pseudocode snippets and steering in sequential order, along with detailed explanations and hyperlinks to sources. For an entire script, you should utilize extra sources, similar to Amazon Q Developer, to construct a totally practical system. The automated workflow described within the publish entails analyzing video content material, producing textual content descriptions, and narrating them utilizing AI voice era. In abstract, whereas highly effective, this requires cautious integration and testing to deploy successfully. By the top of this publish, you’ll perceive the important thing steps, however some extra work is required to create a production-ready resolution to your particular use case.

Answer overview

The next structure diagram demonstrates the end-to-end workflow of the proposed resolution. We are going to describe every element in-depth within the later sections of this publish, however be aware which you can outline the logic inside a single script. You may then run your script on an Amazon Elastic Compute Cloude (Amazon EC2) occasion or in your native pc. For this publish, we assume that you’ll run the script on an Amazon SageMaker notebook.

Companies used

The companies proven within the structure diagram embody:

1. Amazon S3 – Amazon Simple Storage Service (Amazon S3) is an object storage service that gives scalable, sturdy, and extremely accessible storage. On this instance, we use Amazon S3 to retailer the video information (enter) and scene description (textual content information) and audio description (MP3 information) output generated by the answer. The script begins by fetching the supply video from an S3 bucket.
2. Amazon Rekognition – Amazon Rekognition is a pc imaginative and prescient service that may detect and extract video segments or scenes by figuring out technical cues similar to shot boundaries, black frames, and different visible parts. To yield larger accuracy for the generated video descriptions, you utilize Amazon Rekognition to section the supply video into smaller chunks earlier than passing it to Amazon Nova. These video segments may be saved in a short lived listing in your compute machine.
3. Amazon Bedrock – Amazon Bedrock is a managed service that gives entry to massive, pre-trained AI fashions such because the Amazon Nova Professional mannequin, which is used on this resolution to research the content material of every video section and generate detailed scene descriptions. You may retailer these textual content descriptions in a textual content file (for instance, video_analysis.txt).
4. Amazon Polly – Amazon Polly is a text-to-speech service that’s used to transform the textual content descriptions generated by the Amazon Nova Professional mannequin into high-quality audio, made accessible utilizing an MP3 file.

Stipulations

To comply with together with the answer outlined on this publish, it is best to have the next in place:

You should use AWS SDK to create, configure, and handle AWS companies. For Boto3, you’ll be able to embody it on the high of your script utilizing: import boto3

Moreover, you want a mechanism to separate movies. For those who’re utilizing Python, we advocate the moviepy library.
import moviepy # pip set up moviepy

Answer walkthrough

The answer contains the next fundamental steps, which you should utilize as a fundamental construction and customise or increase to suit your use case.

1. Outline the necessities for the AWS atmosphere, together with defining the usage of the Amazon Nova Professional mannequin for its visible help and the AWS Region you’re working in. For optimum throughput, we advocate utilizing inference profiles when configuring Amazon Bedrock to invoke the Amazon Nova Professional mannequin. Initialize a shopper for Amazon Rekognition, which you utilize for its help of segmentation.

CLASS VideoAnalyzer:
	FUNCTION initialize():
 		Set AWS_REGION to "us-east-1"
 		Set MODEL_ID to "amazon.nova-pro-v1:0"
 		Set chunk_delay to twenty Initialize AWS shoppers (Bedrock and Rekognition)

2. Outline a operate for detecting segments within the video. Amazon Rekognition helps segmentation, which implies customers have the choice to detect and extract totally different segments or scenes inside a video. Through the use of the Amazon Rekognition Segment API, you’ll be able to carry out the next:
   1. Detect technical cues similar to black frames, coloration bars, opening and finish credit, and studio logos in a video.
   2. Detect shot boundaries to determine the beginning, finish, and period of particular person pictures throughout the video.

The answer makes use of Amazon Rekognition to partition the video into a number of segments and carry out Amazon Nova Professional-based inference on every section. Lastly, you’ll be able to piece collectively every section’s inference output to return a complete audio description for the complete video.

FUNCTION get_segment_results(job_id):
 	TRY:
 	   Initialize empty segments record 
 	   WHILE extra outcomes exist:
 	         Get section detection outcomes 
                Add segments to record 
                IF no extra outcomes THEN break
          RETURN segments 
       CATCH any errors and return null 

FUNCTION extract_scene(video_path, start_time, end_time):
       TRY: 
           Load video file 
           Validate time vary
           Create non permanent listing 
           Extract video section 
           Save section to file 
           RETURN path to saved section 
       CATCH any errors and return null

Within the previous picture, there are two scenes: a screenshot of 1 scene on the left adopted by the scene that instantly follows it on the precise. With the Amazon Rekognition segmentation API, you’ll be able to determine that the scene has modified—that the content material that’s displayed on display screen is totally different—and subsequently you might want to generate a brand new scene description.

3. Create the segmentation job and:
   • Add the video file for which you need to create an audio description to Amazon S3.
   • Begin the job utilizing that video.

Setting SegmentType=[‘SHOT’] identifies the beginning, finish, and period of a scene. Moreover, MinSegmentConfidence units the minimal confidence Amazon Rekognition should have to return a detected section, with 0 being lowest confidence and 100 being highest.

4. Use the analyze_chunk operate. This operate defines the primary logic of the audio description resolution. Some objects to notice about analyze_chunk:
   • For this instance, we despatched a video scene to Amazon Nova Professional for an evaluation of the contents utilizing the immediate Describe what is going on on this video intimately. This immediate is comparatively easy and experimentation or customization to your use case is inspired. Amazon Nova Professional then returned the textual content description for our video scene.
   • For longer movies with many scenes, you may encounter throttling. That is resolved by implementing a retry mechanism. For particulars on throttling and quotas for Amazon Bedrock, see Quotas for Amazon Bedrock.

FUNCTION analyze_chunk(chunk_path): 
     TRY: 
        Convert video chunk to base64 
        Create request physique for Bedrock 
        Set max_retries and backoff_time 

        WHILE retry_count < max_retries:
          TRY:
             Ship InvokeModel request to Bedrock
             RETURN evaluation outcomes 
          CATCH throttling: 
              Wait and retry with exponential backoff 
          CATCH different errors: 
              Return null 
     CATCH any errors:
         Return null

In impact, the uncooked scenes are transformed into wealthy, descriptive textual content. Utilizing this textual content, you’ll be able to generate a whole scene-by-scene walkthrough of the video and ship it to Amazon Polly for audio.

5. Use the next code to orchestrate the method:
   1. Provoke the detection of the varied segments through the use of Amazon Rekognition.
   2. Every section is processed by a circulate of:
      1. Extraction.
      2. Evaluation utilizing Amazon Nova Professional.
      3. Compiling the evaluation right into a video_analysis.txt file.
6. The analyze_video operate brings collectively all of the parts and produces a textual content file that accommodates the whole, scene-by-scene evaluation of the video contents, with timestamps

FUNCTION analyze_video(video_path, bucket): 
     TRY: 
         Begin section detection 
         Look ahead to job completion 
         Get segments 
         FOR every section: 
             Extract scene 
             Analyze chunk 
             Save evaluation outcomes 
         Write outcomes to file 
      CATCH any errors

For those who refer again to the earlier screenshot, the output—with none extra refinement—will look much like the next picture.

“Phase 103.136-126.026 seconds:
[{'text': 'The video shows a close-up of a coffee cup with steam rising from it, followed by three cups of coffee on a table with milk and sugar jars. A person then picks up a bunch of coffee beans from a plant.'}]
Phase 126.059-133.566 seconds:
[{'text': "The video starts with a person's hand, covered in dirt and holding a branch with green leaves and berries. The person then picks up some berries. The video then shows a man standing in a field with trees and plants. He is holding a bunch of red fruits in his right hand and looking at them. He is wearing a shirt and has a mustache. He seems to be picking the fruits. The fruits are probably coffee beans. The area is surrounded by green plants and trees."}]” 

The next screenshot is an instance is a extra in depth have a look at the video_analysis.txt for the espresso.mp4 video:

7. Ship the contents of the textual content file to Amazon Polly. Amazon Polly provides a voice to the textual content file, finishing the workflow of the audio description resolution.

FUNCTION generate_audio(text_file, output_audio_file):
     TRY:
        Learn evaluation textual content
        Set max_retries and backoff_time

        WHILE retry_count < max_retries:
           TRY:
              Initialize Polly shopper
              Convert textual content to speech
              Save audio file
              RETURN success
           CATCH throttling:
              Wait with exponential backoff
              retry_count += 1
           CATCH different errors:
              retry_count += 1
              Proceed or Break primarily based on error sort
     CATCH any errors:
         RETURN error

For a listing of various voices that you should utilize in Amazon Polly, see Available voices within the Amazon Polly Developer Information.

Your ultimate output with Polly ought to sound one thing like this:

Clear up

It’s a greatest follow to delete the sources you provisioned for this resolution. For those who used an EC2 or SageMaker Pocket book Occasion, cease or terminate it. Keep in mind to delete unused information out of your S3 bucket (eg: video_analysis.txt and video_analysis.mp3).

Conclusion

Recapping the answer at a excessive degree, on this publish, you used:

• Amazon S3 to retailer the unique video, intermediate knowledge, and the ultimate audio description artifacts
• Amazon Rekognition to partition the video file into time-stamped scenes
• Pc imaginative and prescient capabilities from Amazon Nova Professional (accessible by Amazon Bedrock) to research the contents of every scene

We confirmed you learn how to use Amazon Polly to create an MP3 audio file from the ultimate scene description textual content file, which is what might be consumed by the viewers members. The answer outlined on this publish demonstrates learn how to totally automate the method of making audio descriptions for video content material to enhance accessibility. Through the use of Amazon Rekognition for video segmentation, the Amazon Nova Professional mannequin for scene evaluation, and Amazon Polly for text-to-speech, you’ll be able to generate a complete audio description monitor that narrates the important thing visible parts of a video. This end-to-end automation can considerably cut back the time and value required to make video content material accessible for visually impaired audiences, serving to companies and organizations meet their accessibility objectives. With the ability of AWS AI companies, this resolution gives a scalable and environment friendly means to enhance accessibility and inclusion for video-based media.

This resolution isn’t restricted to utilizing it for TV exhibits and films. Any visible media that requires accessibility generally is a candidate! For extra details about the brand new Amazon Nova mannequin household and the superb issues these fashions can do, see Introducing Amazon Nova foundation models: Frontier intelligence and industry leading price performance.

Along with the steps described on this publish, extra actions you may must take embody:

• Eradicating a video section evaluation’s introductory textual content from Amazon Nova. When Amazon Nova returns a response, it’d start with one thing like “On this video…” or one thing related. You most likely need simply the video description itself with out this introductory textual content. If there may be introductory textual content in your scene descriptions, then Amazon Polly will converse it aloud and impression the standard of your audio transcriptions. You may account for this in a couple of methods.
  • For instance, previous to sending it to Amazon Polly, you’ll be able to modify the generated scene descriptions by programmatically eradicating that sort of textual content from them.
  • Alternatively, you should utilize immediate engineering to request that Amazon Bedrock return solely the scene descriptions in a structured format or with none extra commentary.
  • The third choice is to outline and use a instrument when performing inference on Amazon Bedrock. This generally is a extra complete strategy of defining the format of the output that you really want Amazon Bedrock to return. Utilizing instruments to form mannequin output, is called operate calling. For extra info, see Use a tool to complete an Amazon Bedrock model response.
• You also needs to be conscious of the architectural parts of the answer. In a manufacturing atmosphere, being conscious of any potential scaling, safety, and storage parts is necessary as a result of the structure may start to resemble one thing extra complicated than the essential resolution structure diagram that this publish started with.

Concerning the Authors

Dylan Martin is an AWS Options Architect, working primarily within the generative AI area serving to AWS Technical Subject groups construct AI/ML workloads on AWS. He brings his expertise as each a safety options architect and software program engineer. Exterior of labor he enjoys motorcycling, the French Riviera and finding out languages.

Ankit Patel is an AWS Options Developer, a part of the Prototyping And Buyer Engineering (PACE) workforce. Ankit helps prospects carry their progressive concepts to life by fast prototyping; utilizing the AWS platform to construct, orchestrate, and handle customized functions.