Prepare customized pc imaginative and prescient defect detection mannequin utilizing Amazon SageMaker

On October 10, 2024, Amazon introduced the discontinuation of the Amazon Lookout for Vision service, with a scheduled shut down date of October 31, 2025 (see Exploring alternatives and seamlessly migrating data from Amazon Lookout for Vision weblog submit). As a part of our transition steering for patrons, we advocate the usage of Amazon SageMaker AI instruments to construct functions for patrons who’re excited about AI/ML pc imaginative and prescient fashions for automated high quality inspection use circumstances. To assist that effort, AWS has made a pre-trained computer vision defect detection model accessible on AWS Market that may be fine-tuned utilizing Amazon SageMaker AI for a buyer’s particular use case. If run within the cloud, this mannequin solely requires paying for infrastructure prices for coaching or inference. This method supplies the instruments to speed up resolution growth whereas facilitating full flexibility to construct an answer that integrates with any current {hardware} and software program infrastructure.

On this weblog submit, you’ll discover ways to migrate your pc imaginative and prescient workloads from Amazon Lookout for Imaginative and prescient to Amazon SageMaker AI by following our step-by-step steering.

AWS is sharing the main underlying models used for the service to finish customers within the AWS Market. You need to use the 2 principal forms of fashions, binary classification and semantic segmentation, while you practice in your individual AWS accounts for deployment on AWS or on the edge.

This mannequin helps prospects proceed to make use of AWS defect detection know-how at their very own tempo with higher flexibility. For instance, you possibly can practice your fashions with bigger occasion sorts for quicker coaching instances. With entry to set hyperparameters, it’s also possible to alter mannequin conduct that was not beforehand accessible on the AWS console. For instance, you possibly can set the multi-head mannequin for semantic segmentation to disable the binary classifier head. This will make the mannequin mode extra tolerant of fixing background and lighting situations. You may as well personalize the utmost coaching time, which was set to a non-changeable 24-hour restrict on Amazon Lookout for Imaginative and prescient (L4V).

The GitHub repository for Amazon Lookout for Vision has been up to date with a Jupyter Pocket book that will help you practice datasets with these two mannequin sorts and package deal them up. From there you possibly can deploy the fashions through the use of a SageMaker endpoint, or edge units.

To label the pictures past the pattern information, you need to use Amazon SageMaker Ground Truth to allow crowdsourcing or enable non-public groups to label the information, or use a companion resolution resembling Edge Impulse, Roboflow, or SuperbAI to take action. When you’ve got the manifest file of the labeled information, {the marketplace} fashions can be utilized for coaching. You’ll lose a thumbnail-based dataset administration software just like the Amazon Lookout for Imaginative and prescient console, so take into account one of many beforehand talked about companion options to assist handle datasets. You may as well export your current information from the Lookout For Imaginative and prescient service utilizing this information.

Conditions

Earlier than you start, be sure you have the next parts and permissions in place:

• Amazon SageMaker Studio or Amazon SageMaker Unified Studio for built-in growth atmosphere (IDE)
• AWS Identification and Entry Administration (IAM) position with these permissions to observe the precept of least privilege
  • Amazon S3
    • s3:GetObject
    • s3:PutObject
    • s3:DeleteObject
    • s3:ListBucket
  • SageMaker
    • sagemaker:CreateTrainingJob
    • sagemaker:CreateModel
    • sagemaker:CreateEndpoint
    • sagemaker:CreateEndpointConfig
    • sagemaker:CreateTransformJob
    • sagemaker:DescribeTrainingJob
    • sagemaker:DescribeModel
    • sagemaker:DescribeEndpoint
    • sagemaker:DescribeEndpointConfig
    • sagemaker:DescribeTransformJob
    • sagemaker:InvokeEndpoint
    • sagemaker:DeleteEndpoint
    • sagemaker:DeleteEndpointConfig
    • sagemaker:DeleteModel
• Mannequin subscription:
  • An AWS account with a subscription to Computer Vision Defect Detection Model or
  • An IAM position with these three permissions permission to make AWS Market subscriptions within the AWS account you employ:
    • aws-marketplace:ViewSubscriptions
    • aws-marketplace:Unsubscribe
    • aws-marketplace:Subscribe
• Labeled information (you need to use the cookie information pattern in Github) or label your individual information with SageMaker Floor Fact or an AWS Accomplice software
• Primary data of making a SageMaker pocket book occasion and operating Jupyter pocket book

Structure overview

The next diagram illustrates the end-to-end movement, from picture acquisition to inferencing on the edge. This weblog deal with steps 2 and three.

1. Use an edge utility to configure cameras or sensors and seize coaching photographs.
2. Use SageMaker GroundTruth or AWS Accomplice platforms to export and label photographs.
3. Use Amazon SageMaker AI for mannequin coaching.
4. Use REST, PLC, or digital enter for picture acquisition and processing.
5. Run real-time inference utilizing the skilled and deployed mannequin.
6. Publish inference outcomes to analytics and monitoring for alerts and analytics.
7. Carry out automated motion on the machine of concern or notify plant personnel of anomalies from inspection station element utilizing OPC-UA or digital output.
8. Line operators and plant managers obtain notifications for motion.

Arrange the labeling course of

This part covers the steps to arrange the labeling course of utilizing Amazon SageMaker Floor Fact, together with creating a non-public labeling workforce and configuring the labeling job.

1. Configure Amazon SageMaker Floor Fact non-public workforce:
   1. Choose Amazon SageMaker AI, Floor Fact, Labeling workforces.
   2. Choose Personal, then Create Personal Crew.
   3. Enter a workforce identify.
   4. Depart different values as their defaults.
   5. Choose Create a brand new Amazon Cognito person group.
   6. Choose Create non-public Crew.
2. On the Staff tab, choose Invite New Staff.
3. Enter your workforce members’ e mail addresses to ship sign-up invites.

Label the dataset

After efficiently finishing the workforce setup for labelling, the subsequent step is to label the dataset. This part explains methods to put together the dataset by importing the pictures to an Amazon Easy Storage Service (Amazon S3) bucket, then create and run the SageMaker Floor Fact labeling job to label the pictures as regular or anomaly.

1. Add the picture datasets to an Amazon S3 bucket that SageMaker Floor Fact can entry. In the event you don’t have a dataset, you need to use both the cookie-dataset or aliens-dataset.
   • Copy all the photographs from “regular” and “anomaly” folders right into a single listing for SMGT to entry or you’re going to get an error message on the subsequent step.
   • To make use of AWS CloudShell, run the next script:

     #!/bin/bash
     # Clone the repository
     git clone https://github.com/aws-samples/amazon-lookout-for-vision.git
     cd amazon-lookout-for-vision/aliens-dataset
     # Take away current all listing if it exists
     rm -rf all
     # Create a brand new all listing
     mkdir -p all
     # Copy regular photographs to all listing
     cp regular/*.png all/
     # Be sure that we're in the proper listing earlier than operating the loop
     cd "$(dirname "$0")/amazon-lookout-for-vision/aliens-dataset"
     # Copy anomaly photographs with .anomaly.png suffix
     for file in anomaly/*.png; do
       if [ -f "$file" ]; then
         filename=$(basename "$file")
         cp "$file" "all/${filename}.anomaly.png"
       fi
     completed
     # Depend information to confirm
     echo "Regular photographs: $(discover regular -name "*.png" | wc -l)"
     echo "Anomaly photographs: $(discover anomaly -name "*.png" | wc -l)"
     echo "Whole photographs in all listing: $(discover all -type f | wc -l)"
     # Add to S3
     aws s3 cp all/ s3://<BUCKET_NAME>/aliens-dataset-all/ --recursive
     # Clear up - take away the cloned repository
     cd ../..
     rm -rf amazon-lookout-for-vision

   • Alternatively, in case you have the AWS CLI put in, you possibly can copy them with the next instructions (See setting up AWS CLI for a way to do that):

     sh-4.2$ git checkout https://github.com/aws-samples/amazon-lookout-for-vision.git                                     
     sh-4.2$ cd aliens-dataset ## bear in mind the filenames right here conflict, the next Linux command can assist repair this
     sh-4.2$ mkdir all
     sh-4.2$ cp regular/.png all
     sh-4.2$ aws s3 cp s3://aws-blogs-artifacts-public/artifacts/ML-19308/copy_conflicts.sh .
     
     sh-4.2$ bash copy_conflicts.sh
     
     sh-4.2$ ls -al all/
     
     -rwxrwxr-x 1 ec2-user ec2-user 120035 Feb 17 16:39 59.png
     -rwxrwxr-x 1 ec2-user ec2-user 93407 Feb 17 16:39 5.png
     -rwxrwxr-x 1 ec2-user ec2-user 125477 Feb 17 16:39 5.png.anomaly.png
     -rwxrwxr-x 1 ec2-user ec2-user 123679 Feb 17 16:39 60.png
     -rwxrwxr-x 1 ec2-user ec2-user 96330 Feb 17 16:39 6.png
     -rwxrwxr-x 1 ec2-user ec2-user 126014 Feb 17 16:39 6.png.anomaly.png
     -rwxrwxr-x 1 ec2-user ec2-user 81051 Feb 17 16:39 7.png
     -rwxrwxr-x 1 ec2-user ec2-user 128985 Feb 17 16:39 7.png.anomaly.png
     -rwxrwxr-x 1 ec2-user ec2-user 94216 Feb 17 16:39 8.png
     -rwxrwxr-x 1 ec2-user ec2-user 128002 Feb 17 16:39 8.png.anomaly.png
     -rwxrwxr-x 1 ec2-user ec2-user 110814 Feb 17 16:39 9.png
     -rwxrwxr-x 1 ec2-user ec2-user 131385 Feb 17 16:39 9.png.anomaly.png
     
     sh-4.2$aws s3 cp all/ s3://<BUCKET_NAME>/aliens-dataset-all/ --recursive

     Notice: To forestall filename conflict from the 2 folders, a suffix anomaly was added. The uploaded information must be in your <BUCKET_NAME>/aliens-dataset-all bucket for the Floor Fact job.

2. Within the AWS Console, navigate to Amazon SageMaker AI, Floor Fact, Labeling Jobs, Create labeling job.

3. There are a number of choices right here to fill in; crucial fields to fill or choose are:
   • Enter information setup: Choose Automated information setup
   • S3 location for enter datasets: <Full path the place your dataset exists>
   • S3 location information output datasets: <Identical location as enter dataset>
   • Knowledge sort: Choose Picture
   • IAM Function – Choose Create new position should you wouldn’t have one set as much as enable Floor Fact to work together with SageMaker companies.
4. Select Full information setup. An Enter information connection profitable message shows. In the event you get an error, test your IAM position to ensure S3 entry is enabled, and the listing has picture information in it, because it is not going to recurse by means of sub-directories.

5. Choose the duty sort. These fashions assist Picture Classification (Single Label), which is binary classification (suppose good or dangerous), or Semantic segmentation. You can not use a bounding field sort with these fashions. You possibly can change your choice later.
6. Select Subsequent.
7. For Employee sorts, choose Personal. You possibly can learn extra about Amazon Mechanical Turks or labeling subscriptions within the Developer Guide.
8. Beneath Personal groups, choose the non-public workforce you created within the earlier steps.
9. For Job timeout and Job expiration time, go away the default values.
10. Depart Allow automated information labeling unselected. You possibly can learn extra about automated information labeling here; nevertheless, it isn’t appropriate with semantic segmentation.
11. On the Picture classification display, add two new labels: regular and anomaly. You possibly can fill in the remaining as wanted. Select Preview to see a preview of what it would seem like to the top person.
12. Select Create.
13. Choose Floor Fact, after which choose the Personal tab.

14. Open the labeling portal sign-in URL in a brand new tab in your browser after which register to see your assigned duties.
15. Choose an assigned process and select Begin working to label the information.
16. Choose regular or anomaly.

17. When the job is full, make observe of the output dataset location. You will have this for the coaching step.

18. If that you must add staff to the labelling job:
    1. On the Amazon SageMaker AI Floor Fact web page, choose Labeling workforces.
    2. Choose the Personal tab.
    3. Click on on the non-public workforce that was created earlier (CV-team).
    4. Choose the Staff tab
    5. Choose the specified employee from the record and select Add staff to workforce.

• 6. You’ll then be redirected to the Amazon SageMaker AI, labelling workforces web page with a affirmation message that employee has been added.

After you full the labeling process, the output of the duty is used to coach the Pc Imaginative and prescient Detection mannequin from the AWS Market.

Prepare the mannequin

This part discusses coaching the pc imaginative and prescient mannequin utilizing the AWS Market Pc Imaginative and prescient Detection mannequin and the labeled dataset from the earlier step.

1. Go to the AWS Market to subscribe to the mannequin, https://aws.amazon.com/marketplace/pp/prodview-j72hhmlt6avp6.
2. Select Proceed to Subscribe.
3. Select Proceed to configuration.
4. Choose the newest software program model, your Area, and ensure Create a coaching job is chosen.

Notice: Copy the Product Arn and retailer in a textual content editor or notepad for later use.

5. Go to SageMaker AI, Pocket book cases, Create pocket book occasion.

Notice: GPU-enabled pocket book occasion just isn’t required. Amazon SageMaker Coaching jobs will spin up the GPU cases wanted throughout coaching, so most simple cases might be ample.

6. Choose m5.2xl occasion, Jupyter lab 4, with quantity measurement of 128 GB. The default is 5 GB, which is simply too small.
7. Choose an IAM position to permit the pocket book to entry assets in your account. You will have entry to S3.
8. Within the Git Repositories – optionally available part, choose Clone a public Git repository to this pocket book occasion solely.
9. Enter the Git repository URL. Depart all the opposite fields as their default, then select Create pocket book occasion to start out the occasion.
10. After the occasion begins, (the standing will show as InService), choose Open JupyterLab motion for the brand new pocket book occasion.

JupyterLab opens:

11. On the left navigation pane, open the computer-vision-defect-detection folder.

12. Within the AWS Console, go to Market, Handle subscriptions, after which copy the ARN of your mannequin subscription.

13. Within the Jupyter pocket book, find the snippet under and replace the placeholder worth for algorithm_name variable with the Product Arn you copied within the earlier step.

# TODO: change this to make use of subscribed SageMaker algorithm algorithm_name = "<Buyer to specify the algorithm identify after subscription >"

The bucket that may be used for this step could be routinely created and named within the format SageMaker-<REGION>-<ACCOUNT_ID>.

# Initialize SageMaker session and get execution position
sagemaker_session = sagemaker.Session()
area = sagemaker_session.boto_region_name
#bucket = sagemaker_session.default_bucket()
position = get_execution_role()
# Undertaking identify could be used as a part of s3 output path
venture = "ComputerVisionDefectDetection”

14. Within the AWS Console, navigate to Amazon SageMaker AI, Floor Fact, Labeling jobs and choose the job that was accomplished.
15. Establish and be aware of the output photographs folder (Output dataset location)

Notice: To begin the coaching job, have a look at the trail for the output manifest in <BUCKET NAME>/aliens-dataset/all/aliensv2/manifests/output/output.manifest—this would be the coaching manifest for the subsequent step.

16. Set the bucket variable to be the pictures bucket identify that you simply beforehand set and object key the trail to your manifest:
    • bucket: the place to retailer the manifest file
    • classification_manifest_key: the place the output manifest file is saved (for instance, aliens-dataset-all/[job-name]/manifests/output/output.manifest)
17. Evaluation the mannequin coaching configuration within the Classification Mannequin with Algorithm Estimator part.

# Create AlgorithmEstimator for classificatio
classification_estimator = AlgorithmEstimator(
algorithm_arn=algorithm_name,
position=position, instance_count=1,
instance_type="ml.g4dn.2xlarge",
volume_size=20, max_run=7200,
input_mode="Pipe", # REQUIRED: Algorithm solely helps Pipe mode
sagemaker_session=sagemaker_session,
enable_network_isolation=True
)

# Set hyperparameters
classification_estimator.set_hyperparameters(
ModelType="classification",
TestInputDataAttributeNames="source-ref,anomaly-label-metadata,anomaly-label", 
TrainingInputDataAttributeNames="source-ref,anomaly-label-metadata,anomaly-label")

print("Classification estimator configured efficiently")</code></pre><pre><code class="lang-python"># Outline coaching enter utilizing TrainingInput class
classification_training_input = TrainingInput(
s3_data=classification_s3_path, '
s3_data_type="AugmentedManifestFile",
attribute_names=[
'source-ref',
'anomaly-label-metadata',
'anomaly-label'
],
record_wrapping='RecordIO',
input_mode="Pipe" # Should match the estimator's input_mode)
# Begin coaching job
classification_job_name = f'defect-detection-classification-
{datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")}
'print(f"Beginning classification coaching job: {classification_job_name}")
classification_estimator.match(
inputs={'coaching': classification_training_input},
job_name=classification_job_name,
wait=True,
logs=True

)

Notice: The job makes use of NVIDIA G4DN cases. They are often sized as much as a bigger occasion to lower coaching time, however on a solely 118 cases. The picture dataset coaching finishes in lower than 10 minutes with a g4dn.2xl. You possibly can experiment with different occasion sorts, nevertheless outcomes could range as a result of the fashions had been extensively examined on the G4DN cases.

18. Validate the values of TestInputDataAttributeNames and TrainingInputDataAttributeNames within the Hyperparameters part, in addition to AttributeNames within the

TrainingInput part. The labels on all three should match the construction of your manifest file. Here’s a pattern manifest:

{
    "source-ref": "s3://[bucketname]/getting-started/training-images/anomaly-1.jpg",
    "anomaly-label-metadata": {
        "job-name": "anomaly-label",
        "class-name": "anomaly",
        "human-annotated": "sure",
        "creation-date": "2022-08-22T20:52:51.851Z",
        "sort": "groundtruth/image-classification"
    },
    "anomaly-label": 1
}
{
    "source-ref": "s3://[bucketname]/getting-started/training-images/anomaly-2.jpg",
    "anomaly-label-metadata": {
        "job-name": "anomaly-label",
        "class-name": "anomaly",
        "human-annotated": "sure",
        "creation-date": "2022-08-22T21:11:39.545Z",
        "sort": "groundtruth/image-classification"
    },
    "anomaly-label": 1
}

Notice: Two of the three values embody the labelling job identify.

response = sagemaker.create_training_job(
    TrainingJobName=classification_training_job_name,
    HyperParameters={
        'ModelType': 'classification',
        'TestInputDataAttributeNames': 'source-ref,aliens-v3,aliens-v3-metadata',
        'TrainingInputDataAttributeNames': 'source-ref,aliens-v3,aliens-v3-metadata'
    }
)

19. Run all of the cells or blocks listed within the Classification Mannequin with Algorithm Estimator part to start out the coaching job.
20. If you wish to practice a segmentation mannequin as effectively, observe the steps within the Segmentation Mannequin with Algorithm Estimator part.

Notice: After the coaching is accomplished, you’re prepared to check it!  There are few inference choices accessible for this:

• Actual-time inference utilizing Amazon SageMaker endpoints
• Amazon SageMaker AI Batch Remodel inference.
• Edge deployment

Deploy the mannequin

Amazon SageMaker AI endpoints and Amazon SageMaker AI Batch Remodel inference are each used for inference however serve totally different functions.

Amazon SageMaker AI endpoints

Amazon SageMaker AI endpoints are used for real-time inference, offering low-latency predictions appropriate for functions requiring rapid responses. Endpoints stay energetic whereas they’re deployed, making them higher suited to steady and regular site visitors, however probably extra expensive attributable to ongoing useful resource utilization.

1. Within the Jupyter pocket book, navigate to the (Elective) Working real-time inference utilizing Amazon SageMaker endpoints part.
2. Run the next cell blocks to arrange and invoke the endpoint:

#classification_training_job_name = "defect-detection-classification-2025-10-01-00-29-57" # take away

classification_training_job_name = "<present coaching job identify right here>"

# Create estimator from coaching job
estimator = AlgorithmEstimator.connect(classification_training_job_name)

# Deploy endpoint utilizing SageMaker v2 SDK
predictor = estimator.deploy(
    initial_instance_count=1,
    instance_type="ml.c5.2xlarge"
)

print(f"Endpoint deployed: {predictor.endpoint_name}")

#Invoke the endpoint 

    # Invoke the endpoint utilizing predictor
    end result = predictor.predict(image_data)
    
    # Clear up the momentary file
    os.take away(local_file)
    
    # Print the end result
    print("nEndpoint Response:")
    print(json.dumps(end result, indent=2))

3. Validate the inference, then delete the endpoint by operating the next block:

# Delete the endpoint

predictor.delete_endpoint()
print("Endpoint deleted")

Notice: In the event you begin an endpoint, bear in mind you can be billed whereas it’s operating till you flip it off.

Amazon SageMaker AI Batch Remodel

Batch Remodel is designed for offline inference and making predictions on massive datasets saved in S3, and is good for bulk processing the place low latency just isn’t important. After the job is full, the assets are launched, making it cost-effective for sporadic workloads.

1. Navigate to the (Elective) Run Batch Remodel Inference utilizing SageMaker SDK v2 part.
2. Outline the s3_input_data and s3_output_path parameters.

# Run batch remodel job

#############################################
# Change to your enter/output information S3 path  #
#############################################

s3_input_data = "s3://<Specify-s3-path-to-test-images>"
s3_output_path = f"s3://{bucket}/{venture}/batch-transform-output"

3. Run all of the cells and blocks within the (Elective) Run Batch Remodel Inference utilizing SageMaker SDK v2 part to finish the batch inference.
4. Validate the batch remodel job after completion by navigating to the s3_output_path folder. The next is a pattern inference output file:

{
    "Supply": {
        "Sort": "direct"
    },
    "IsAnomalous": true,
    "Confidence": 0.92744799389183
}

Clear up

To keep away from incurring pointless costs, delete the next assets while you now not want them:

1. Delete SageMaker endpoints.
   1. Navigate to the Amazon SageMaker Console.
   2. Choose Endpoints.
   3. Choose the endpoint you created.
   4. Select Delete.
2. Delete SageMaker Pocket book cases.
   1. Navigate to the Amazon SageMaker Console.
   2. Choose Pocket book cases.
   3. Choose the pocket book occasion you created.
   4. Select Cease if the occasion is operating.
   5. As soon as stopped, select Delete.
3. Delete S3 objects and buckets.
   1. Navigate to the Amazon S3 Console.
   2. Delete all objects within the buckets you created for this tutorial.
   3. Delete the empty buckets.
4. Delete the Floor Fact labeling workforce.
   1. Navigate to Floor Fact.
   2. Choose Labeling workforces.
   3. Choose the Personal tab.
   4. Choose the non-public workforce you created.
   5. Select Delete workforce.

Conclusion

On this weblog submit, we’ve demonstrated methods to transition from Amazon Lookout for Imaginative and prescient to utilizing the underlying Pc Imaginative and prescient Detection fashions accessible by means of the AWS Market, exhibiting the step-by-step means of organising labeling, coaching the mannequin, and operating inference by means of batch transformation. The transition supplies prospects with higher flexibility by way of coaching choices, hyperparameter changes, and deployment selections whereas persevering with to make use of AWS defect detection know-how at their very own tempo. Additionally remember to take a look at our edge-based open supply built-in Defect Detection Application on GitHub if you need to mix what you’ve got discovered right here.

In regards to the authors

Ryan Vanderwerf is a is a senior companion options architect at Amazon Internet Providers specializing in good manufacturing, imaginative and prescient, and machine studying. Ryan beforehand offered Java digital machine-focused consulting and venture growth as a software program engineer at OCI on the Grails and Micronaut workforce. He was chief architect/director of merchandise at ReachForce, with a deal with software program and system structure for AWS Cloud SaaS options for advertising information administration. Ryan has constructed a number of SaaS options in a number of domains resembling monetary, media, telecom, and e-learning corporations since 1996

Lu Min is a Software program Growth Engineer for AWS Edge ML companies, centered on growing machine studying options that function on the edge for AWS prospects. With experience in optimizing ML fashions for resource-constrained environments, Lu helps prospects implement environment friendly inference capabilities on edge units and cloud communication, in addition to handle mannequin lifecycle utilizing AWS SageMaker.

Tim Westman is the Product Supervisor and Go-to-Market Lead for Edge Machine Studying, AWS. Tim leads the Product Administration and Enterprise Growth for the Edge Machine Studying enterprise at Amazon Internet Providers. On this position, he works with prospects to assist construct pc imaginative and prescient options on the edge to unravel complicated operational challenges. Tim has greater than 30 years of expertise in gross sales, enterprise growth and product administration roles for main {hardware} and software program corporations, with the final 8 years specializing in AI and pc imaginative and prescient for IoT functions.

Kunle Adeleke is an enterprise options architect, offering steering to massive AWS business prospects in numerous industries craft their know-how technique. Kunle has led enterprise structure groups and software program growth groups in each authorities and business sectors. His deep experience spans software program growth, resolution structure, enterprise structure, safety, and information & AI/ML.