Bundle and deploy classical ML and LLMs simply with Amazon SageMaker, half 1: PySDK Enhancements

Amazon SageMaker is a totally managed service that permits builders and information scientists to rapidly and effortlessly construct, prepare, and deploy machine studying (ML) fashions at any scale. SageMaker makes it simple to deploy fashions into manufacturing immediately by means of API calls to the service. Fashions are packaged into containers for sturdy and scalable deployments. Though it supplies varied entry factors just like the SageMaker Python SDK, AWS SDKs, the SageMaker console, and Amazon SageMaker Studio notebooks to simplify the method of coaching and deploying ML fashions at scale, clients are nonetheless in search of higher methods to deploy their fashions for playground testing and to optimize manufacturing deployments.

We’re launching two new methods to simplify the method of packaging and deploying fashions utilizing SageMaker.

On this put up, we introduce the brand new SageMaker Python SDK ModelBuilder expertise, which goals to attenuate the training curve for brand spanking new SageMaker customers like information scientists, whereas additionally serving to skilled MLOps engineers maximize utilization of SageMaker internet hosting providers. It reduces the complexity of preliminary setup and deployment, and by offering steerage on greatest practices for profiting from the complete capabilities of SageMaker. We offer detailed data and GitHub examples for this new SageMaker functionality.

The opposite new launch is to make use of the brand new interactive deployment expertise in SageMaker Studio. We focus on this in Half 2.

Deploying fashions to a SageMaker endpoint entails a sequence of steps to get the mannequin able to be hosted on a SageMaker endpoint. This includes getting the mannequin artifacts within the appropriate format and construction, creating inference code, and specifying important particulars just like the mannequin picture URL, Amazon Simple Storage Service (Amazon S3) location of mannequin artifacts, serialization and deserialization steps, and needed AWS Identity and Access Management (IAM) roles to facilitate acceptable entry permissions. Following this, an endpoint configuration requires figuring out the inference sort and configuring respective parameters corresponding to occasion varieties, counts, and visitors distribution amongst mannequin variants.

To additional assist our clients when utilizing SageMaker internet hosting, we launched the brand new ModelBuilder class within the SageMaker Python SDK, which brings the next key advantages when deploying fashions to SageMaker endpoints:

• Unifies the deployment expertise throughout frameworks – The brand new expertise supplies a constant workflow for deploying fashions constructed utilizing completely different frameworks like PyTorch, TensorFlow, and XGBoost. This simplifies the deployment course of.
• Automates mannequin deployment – Duties like deciding on acceptable containers, capturing dependencies, and dealing with serialization/deserialization are automated, decreasing guide effort required for deployment.
• Offers a easy transition from native to SageMaker hosted endpoint – With minimal code modifications, fashions may be simply transitioned from native testing to deployment on a SageMaker endpoint. Dwell logs make debugging seamless.

Total, SageMaker ModelBuilder simplifies and streamlines the mannequin packaging course of for SageMaker inference by dealing with low-level particulars and supplies instruments for testing, validation, and optimization of endpoints. This improves developer productiveness and reduces errors.

Within the following sections, we deep dive into the small print of this new function. We additionally focus on learn how to deploy fashions to SageMaker internet hosting utilizing ModelBuilder, which simplifies the method. Then we stroll you thru a number of examples for various frameworks to deploy each conventional ML fashions and the muse fashions that energy generative AI use circumstances.

Attending to know SageMaker ModelBuilder

The brand new ModelBuilder is a Python class centered on taking ML fashions constructed utilizing frameworks, like XGBoost or PyTorch, and changing them into fashions which might be prepared for deployment on SageMaker. ModelBuilder supplies a construct() perform, which generates the artifacts in accordance the mannequin server, and a deploy() perform to deploy domestically or to a SageMaker endpoint. The introduction of this function simplifies the combination of fashions with the SageMaker surroundings, optimizing them for efficiency and scalability. The next diagram exhibits how ModelBuilder works on a high-level.

ModelBuilder class

The ModelBuilder class present completely different choices for personalisation. Nonetheless, to deploy the framework mannequin, the mannequin builder simply expects the mannequin, enter, output, and function:

class ModelBuilder(
    mannequin, # mannequin id or mannequin object
    role_arn, # IAM function
    schema_builder, # defines the enter and output
    mode, # choose between native deployment and depoy to SageMaker Endpoints
    ...
)

SchemaBuilder

The SchemaBuilder class lets you outline the enter and output on your endpoint. It permits the schema builder to generate the corresponding marshaling features for serializing and deserializing the enter and output. The next class file supplies all of the choices for personalisation:

class SchemaBuilder(
    sample_input: Any,
    sample_output: Any,
    input_translator: CustomPayloadTranslator = None,
    output_translator: CustomPayloadTranslator = None
)

Nonetheless, normally, simply pattern enter and output would work. For instance:

enter = "How is the demo going?"
output = "Remark la démo va-t-elle?"
schema = SchemaBuilder(enter, output)

By offering pattern enter and output, SchemaBuilder can mechanically decide the required transformations, making the combination course of extra simple. For extra superior use circumstances, there’s flexibility to offer customized translation features for each enter and output, making certain that extra complicated information constructions will also be dealt with effectively. We reveal this within the following sections by deploying completely different fashions with varied frameworks utilizing ModelBuilder.

Native mode expertise

On this instance, we use ModelBuilder to deploy XGBoost mannequin domestically. You need to use Mode to change between native testing and deploying to a SageMaker endpoint. We first prepare the XGBoost mannequin (domestically or in SageMaker) and retailer the mannequin artifacts within the working listing:

# Prepare the mannequin
mannequin = XGBClassifier()
mannequin.match(X_train, y_train)
mannequin.save_model(model_dir + "/my_model.xgb")

Then we create a ModelBuilder object by passing the precise mannequin object, the SchemaBuilder that makes use of the pattern check enter and output objects (the identical enter and output we used when coaching and testing the mannequin) to deduce the serialization wanted. Observe that we use Mode.LOCAL_CONTAINER to specify a neighborhood deployment. After that, we name the build perform to mechanically establish the supported framework container picture in addition to scan for dependencies. See the next code:

model_builder_local = ModelBuilder(
    mannequin=mannequin,  
    schema_builder=SchemaBuilder(X_test, y_pred), 
    role_arn=execution_role, 
    mode=Mode.LOCAL_CONTAINER
)
xgb_local_builder = model_builder_local.construct()

Lastly, we are able to name the deploy perform within the mannequin object, which additionally supplies dwell logging for simpler debugging. You don’t must specify the occasion sort or rely as a result of the mannequin will likely be deployed domestically. When you offered these parameters, they are going to be ignored. This perform will return the predictor object that we are able to use to make prediction with the check information:

# notice: all of the serialization and deserialization is dealt with by the mannequin builder.
predictor_local = xgb_local_builder.deploy(
# instance_type="ml.c5.xlarge",
# initial_instance_count=1
)

# Make prediction for check information. 
predictor_local.predict(X_test)

Optionally, it’s also possible to management the loading of the mannequin and preprocessing and postprocessing utilizing InferenceSpec. We offer extra particulars later on this put up. Utilizing LOCAL_CONTAINER is an effective way to check out your script domestically earlier than deploying to a SageMaker endpoint.

Seek advice from the model-builder-xgboost.ipynb instance to check out deploying each domestically and to a SageMaker endpoint utilizing ModelBuilder.

Deploy conventional fashions to SageMaker endpoints

Within the following examples, we showcase learn how to use ModelBuilder to deploy conventional ML fashions.

XGBoost fashions

Much like the earlier part, you’ll be able to deploy an XGBoost mannequin to a SageMaker endpoint by altering the mode parameter when creating the ModelBuilder object:

model_builder = ModelBuilder(
    mannequin=mannequin,  
    schema_builder=SchemaBuilder(sample_input=sample_input, sample_output=sample_output), 
    role_arn=execution_role, 
    mode=Mode.SAGEMAKER_ENDPOINT
)
xgb_builder = model_builder.construct()
predictor = xgb_builder.deploy(
    instance_type="ml.c5.xlarge",
    initial_instance_count=1
)

Observe that when deploying to SageMaker endpoints, it is advisable specify the occasion sort and occasion rely when calling the deploy perform.

Seek advice from the model-builder-xgboost.ipynb instance to deploy an XGBoost mannequin.

Triton fashions

You need to use ModelBuilder to serve PyTorch fashions on Triton Inference Server. For that, it is advisable specify the model_server parameter as ModelServer.TRITON, cross a mannequin, and have a SchemaBuilder object, which requires pattern inputs and outputs from the mannequin. ModelBuilder will maintain the remaining for you.

model_builder = ModelBuilder(
    mannequin=mannequin,  
    schema_builder=SchemaBuilder(sample_input=sample_input, sample_output=sample_output), 
    role_arn=execution_role,
    model_server=ModelServer.TRITON, 
    mode=Mode.SAGEMAKER_ENDPOINT
)

triton_builder = model_builder.construct()

predictor = triton_builder.deploy(
    instance_type="ml.g4dn.xlarge",
    initial_instance_count=1
)

Seek advice from model-builder-triton.ipynb to deploy a mannequin with Triton.

Hugging Face fashions

On this instance, we present you learn how to deploy a pre-trained transformer mannequin offered by Hugging Face to SageMaker. We need to use the Hugging Face pipeline to load the mannequin, so we create a customized inference spec for ModelBuilder:

# customized inference spec with hugging face pipeline
class MyInferenceSpec(InferenceSpec):
    def load(self, model_dir: str):
        return pipeline("translation_en_to_fr", mannequin="t5-small")
        
    def invoke(self, enter, mannequin):
        return mannequin(enter)
    
inf_spec = MyInferenceSpec()

We additionally outline the enter and output of the inference workload by defining the SchemaBuilder object primarily based on the mannequin enter and output:

schema = SchemaBuilder(worth,output)

Then we create the ModelBuilder object and deploy the mannequin onto a SageMaker endpoint following the identical logic as proven within the different instance:

builder = ModelBuilder(
    inference_spec=inf_spec,
    mode=Mode.SAGEMAKER_ENDPOINT,  # you'll be able to change it to Mode.LOCAL_CONTAINER for native testing
    schema_builder=schema,
    image_uri=picture,
)
mannequin = builder.construct(
    role_arn=execution_role,
    sagemaker_session=sagemaker_session,
)
predictor = mannequin.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge"
)

Seek advice from model-builder-huggingface.ipynb to deploy a Hugging Face pipeline mannequin.

Deploy basis fashions to SageMaker endpoints

Within the following examples, we showcase learn how to use ModelBuilder to deploy basis fashions. Similar to the fashions talked about earlier, all that’s required is the mannequin ID.

Hugging Face Hub

If you wish to deploy a basis mannequin from Hugging Face Hub, all it is advisable do is cross the pre-trained mannequin ID. For instance, the next code snippet deploys the meta-llama/Llama-2-7b-hf mannequin domestically. You may change the mode to Mode.SAGEMAKER_ENDPOINT to deploy to SageMaker endpoints.

model_builder = ModelBuilder(
    mannequin="meta-llama/Llama-2-7b-hf",
    schema_builder=SchemaBuilder(sample_input, sample_output),
    model_path="/residence/ec2-user/SageMaker/LoadTestResources/meta-llama2-7b", #native path the place artifacts will likely be saved
    mode=Mode.LOCAL_CONTAINER,
    env_vars={
        # Llama 2 is a gated mannequin and requires a Hugging Face Hub token.
        "HUGGING_FACE_HUB_TOKEN": "<YourHuggingFaceToken>"
 
    }
)
mannequin = model_builder.construct()
local_predictor = mannequin.deploy()

For gated fashions on Hugging Face Hub, it is advisable request entry through Hugging Face Hub and use the related key by passing it because the surroundings variable HUGGING_FACE_HUB_TOKEN. Some Hugging Face fashions could require trusting distant code. It may be set as an surroundings variable as properly utilizing HF_TRUST_REMOTE_CODE. By default, ModelBuilder will use a Hugging Face Textual content Era Inference (TGI) container because the underlying container for Hugging Face fashions. If you want to make use of AWS Giant Mannequin Inference (LMI) containers, you’ll be able to arrange the model_server parameter as ModelServer.DJL_SERVING while you configure the ModelBuilder object.

A neat function of ModelBuilder is the flexibility to run native tuning of the container parameters while you use LOCAL_CONTAINER mode. This function can be utilized by merely operating tuned_model = mannequin.tune().

Seek advice from demo-model-builder-huggingface-llama2.ipynb to deploy a Hugging Face Hub mannequin.

SageMaker JumpStart

Amazon SageMaker JumpStart additionally provides plenty of pre-trained basis fashions. Similar to the method of deploying a mannequin from Hugging Face Hub, the mannequin ID is required. Deploying a SageMaker JumpStart mannequin to a SageMaker endpoint is as simple as operating the next code:

model_builder = ModelBuilder(
    mannequin="huggingface-llm-falcon-7b-bf16",
    schema_builder=SchemaBuilder(sample_input, sample_output),
    role_arn=execution_role
)

sm_ep_model = model_builder.construct()

predictor = sm_ep_model.deploy()

For all obtainable SageMaker JumpStart mannequin IDs, seek advice from Built-in Algorithms with pre-trained Model Table. Seek advice from model-builder-jumpstart-falcon.ipynb to deploy a SageMaker JumpStart mannequin.

Inference element

ModelBulder lets you use the brand new inference element functionality in SageMaker to deploy fashions. For extra data on inference parts, see Reduce Model Deployment Costs By 50% on Average Using SageMaker’s Latest Features. You need to use inference parts for deployment with ModelBuilder by specifying endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED within the deploy() technique. You can too use the tune() technique, which fetches the optimum variety of accelerators, and modify it if required.

resource_requirements = ResourceRequirements(
    requests={
        "num_accelerators": 4,
        "reminiscence": 1024,
        "copies": 1,
    },
    limits={},
)

goldfinch_predictor_2 = model_2.deploy(
    mode=Mode.SAGEMAKER_ENDPOINT,
    endpoint_type=EndpointType.INFERENCE_COMPONENT_BASED,
    ...
	
)

Seek advice from model-builder-inference-component.ipynb to deploy a mannequin as an inference element.

Customise the ModelBuilder Class

The ModelBuilder class lets you customise mannequin loading utilizing InferenceSpec.

As well as, you’ll be able to management payload and response serialization and deserialization and customise preprocessing and postprocessing utilizing CustomPayloadTranslator. Moreover, when it is advisable prolong our pre-built containers for mannequin deployment on SageMaker, you should utilize ModelBuilder to deal with the mannequin packaging course of. On this following part, we offer extra particulars of those capabilities.

InferenceSpec

InferenceSpec provides a further layer of customization. It lets you outline how the mannequin is loaded and the way it will deal with incoming inference requests. By means of InferenceSpec, you’ll be able to outline customized loading procedures on your fashions, bypassing the default loading mechanisms. This flexibility is especially helpful when working with non-standard fashions or customized inference pipelines. The invoke technique may be personalized, offering you with the flexibility to tailor how the mannequin processes incoming requests (preprocessing and postprocessing). This customization may be important to make sure that the inference course of aligns with the precise wants of the mannequin. See the next code:

class InferenceSpec(abc.ABC):
    @abc.abstractmethod
    def load(self, model_dir: str):
        cross

    @abc.abstractmethod
    def invoke(self, input_object: object, mannequin: object):
        cross

The next code exhibits an instance of utilizing this class:

class MyInferenceSpec(InferenceSpec):
    def load(self, model_dir: str):
        return // mannequin object

    def invoke(self, enter, mannequin):
        return mannequin(enter)

CustomPayloadTranslator

When invoking SageMaker endpoints, the info is distributed by means of HTTP payloads with completely different MIME varieties. For instance, a picture despatched to the endpoint for inference must be transformed to bytes on the consumer aspect and despatched by means of the HTTP payload to the endpoint. When the endpoint receives the payload, it must deserialize the byte string again to the info sort that’s anticipated by the mannequin (also referred to as server-side deserialization). After the mannequin finishes prediction, the outcomes should be serialized to bytes that may be despatched again by means of the HTTP payload to the person or consumer. When the consumer receives the response byte information, it must carry out client-side deserialization to transform the bytes information again to the anticipated information format, corresponding to JSON. At a minimal, you need to convert the data for the following (as numbered within the following diagram):

1. Inference request serialization (dealt with by the consumer)
2. Inference request deserialization (dealt with by the server or algorithm)
3. Invoking the mannequin in opposition to the payload
4. Sending response payload again
5. Inference response serialization (dealt with by the server or algorithm)
6. Inference response deserialization (dealt with by the consumer)

The next diagram exhibits the method of serialization and deserialization throughout the invocation course of.

Within the following code snippet, we present an instance of CustomPayloadTranslator when further customization is required to deal with each serialization and deserialization within the consumer and server aspect, respectively:

from sagemaker.serve import CustomPayloadTranslator

# request translator
class MyRequestTranslator(CustomPayloadTranslator):
    # This perform converts the payload to bytes - occurs on consumer aspect
    def serialize_payload_to_bytes(self, payload: object) -> bytes:
        # converts the enter payload to bytes
        ... ...
        return  //return object as bytes
        
    # This perform converts the bytes to payload - occurs on server aspect
    def deserialize_payload_from_stream(self, stream) -> object:
        # convert bytes to in-memory object
        ... ...
        return //return in-memory object
        
# response translator 
class MyResponseTranslator(CustomPayloadTranslator):
    # This perform converts the payload to bytes - occurs on server aspect
    def serialize_payload_to_bytes(self, payload: object) -> bytes:
        # converts the response payload to bytes
        ... ...
        return //return object as bytes
    
    # This perform converts the bytes to payload - occurs on consumer aspect
    def deserialize_payload_from_stream(self, stream) -> object:
        # convert bytes to in-memory object
        ... ...
        return //return in-memory object

Within the demo-model-builder-pytorch.ipynb pocket book, we reveal learn how to simply deploy a PyTorch mannequin to a SageMaker endpoint utilizing ModelBuilder with the CustomPayloadTranslator and the InferenceSpec class.

Stage mannequin for deployment

If you wish to stage the mannequin for inference or within the mannequin registry, you should utilize mannequin.create() or mannequin.register(). The enabled mannequin is created on the service, after which you’ll be able to deploy later. See the next code:

model_builder = ModelBuilder(
    mannequin=mannequin,  
    schema_builder=SchemaBuilder(X_test, y_pred), 
    role_arn=execution_role, 
)
deployable_model = model_builder.construct()

deployable_model.create() # deployable_model.register() for mannequin registry

Use customized containers

SageMaker supplies pre-built Docker images for its built-in algorithms and the supported deep studying frameworks used for coaching and inference. If a pre-built SageMaker container doesn’t fulfill all of your necessities, you’ll be able to prolong the prevailing picture to accommodate your wants. By extending a pre-built picture, you should utilize the included deep studying libraries and settings with out having to create a picture from scratch. For extra particulars about learn how to prolong the pre-built containers, seek advice from SageMaker doc. ModelBuilder helps use circumstances when bringing your personal containers which might be prolonged from our pre-built Docker containers.

To make use of your personal container picture on this case, it is advisable set the fields image_uri and model_server when defining ModelBuilder:

model_builder = ModelBuilder(
    mannequin=mannequin,  # Move within the precise mannequin object. its "predict" technique will likely be invoked within the endpoint.
    schema_builder=SchemaBuilder(X_test, y_pred), # Move in a "SchemaBuilder" which is able to use the pattern check enter and output objects to deduce the serialization wanted.
    role_arn=execution_role, 
    image_uri=image_uri, # REQUIRED FOR BYOC: Passing in picture hosted in private ECR Repo
    model_server=ModelServer.TORCHSERVE, # REQUIRED FOR BYOC: Passing in mannequin server of selection
    mode=Mode.SAGEMAKER_ENDPOINT,
    dependencies={"auto": True, "customized": ["protobuf==3.20.2"]}
)

Right here, the image_uri would be the container picture ARN that’s saved in your account’s Amazon Elastic Container Registry (Amazon ECR) repository. One instance is proven as follows:

# Pulled the xgboost:1.7-1 DLC and pushed to private ECR repo
image_uri = "<your_account_id>.dkr.ecr.us-west-2.amazonaws.com/my-byoc:xgb"

When the image_uri is about, throughout the ModelBuilder construct course of, it would skip auto detection of the picture because the picture URI is offered. If model_server is just not set in ModelBuilder, you’ll obtain a validation error message, for instance:

ValueError: Model_server should be set when image_uri is about. Supported mannequin servers: {<ModelServer.TRITON: 5>, <ModelServer.DJL_SERVING: 4>, <ModelServer.TORCHSERVE: 1>}

As of the publication of this put up, ModelBuilder helps bringing your personal containers which might be prolonged from our pre-built DLC container images or containers constructed with the mannequin servers like Deep Java Library (DJL), Text Generation Inference (TGI), TorchServe, and Triton inference server.

Customized dependencies

When operating ModelBuilder.construct(), by default it mechanically captures your Python surroundings right into a necessities.txt file and installs the identical dependency within the container. Nonetheless, generally your native Python surroundings will battle with the surroundings within the container. ModelBuilder supplies a easy means so that you can modify the captured dependencies to repair such dependency conflicts by permitting you to offer your customized configurations into ModelBuilder. Observe that that is just for TorchServe and Triton with InferenceSpec. For instance, you’ll be able to specify the enter parameter dependencies, which is a Python dictionary, in ModelBuilder as follows:

dependency_config = {
   "auto" = True,
   "necessities" = "/path/to/your/necessities.txt"
   "customized" = ["module>=1.2.3,<1.5", "boto3==1.16.*", "some_module@http://some/url"]
}
  
ModelBuilder(
    # Different params
    dependencies=dependency_config,
).construct()

We outline the next fields:

• auto – Whether or not to attempt to auto seize the dependencies in your surroundings.
• necessities – A string of the trail to your personal necessities.txt file. (That is optionally available.)
• customized – A listing of some other customized dependencies that you simply need to add or modify. (That is optionally available.)

If the identical module is laid out in a number of locations, customized may have highest precedence, then necessities, and auto may have lowest precedence. For instance, let’s say that in autodetect, ModelBuilder detects numpy==1.25, and a necessities.txt file is offered that specifies numpy>=1.24,<1.26. Moreover, there’s a customized dependency: customized = ["numpy==1.26.1"]. On this case, numpy==1.26.1 will likely be picked once we set up dependencies within the container.

Clear up

If you’re achieved testing the fashions, as a greatest apply, delete the endpoint to save lots of prices if the endpoint is now not required. You may observe the Clear up part in every of the demo notebooks or use following code to delete the mannequin and endpoint created by the demo:

predictor.delete_model()
predictor.delete_endpoint()

Conclusion

The brand new SageMaker ModelBuilder functionality simplifies the method of deploying ML fashions into manufacturing on SageMaker. By dealing with lots of the complicated particulars behind the scenes, ModelBuilder reduces the training curve for brand spanking new customers and maximizes utilization for skilled customers. With just some traces of code, you’ll be able to deploy fashions with built-in frameworks like XGBoost, PyTorch, Triton, and Hugging Face, in addition to fashions offered by SageMaker JumpStart into sturdy, scalable endpoints on SageMaker.

We encourage all SageMaker customers to check out this new functionality by referring to the ModelBuilder documentation web page. ModelBuilder is accessible now to all SageMaker customers at no further cost. Make the most of this simplified workflow to get your fashions deployed sooner. We sit up for listening to how ModelBuilder accelerates your mannequin growth lifecycle!

Particular due to Sirisha Upadhyayala, Raymond Liu, Gary Wang, Dhawal Patel, Deepak Garg and Ram Vegiraju.

Concerning the authors

Melanie Li, PhD, is a Senior AI/ML Specialist TAM at AWS primarily based in Sydney, Australia. She helps enterprise clients construct options utilizing state-of-the-art AI/ML instruments on AWS and supplies steerage on architecting and implementing ML options with greatest practices. In her spare time, she likes to discover nature and spend time with household and mates.

Marc Karp is an ML Architect with the Amazon SageMaker Service workforce. He focuses on serving to clients design, deploy, and handle ML workloads at scale. In his spare time, he enjoys touring and exploring new locations.

Sam Edwards, is a Cloud Engineer (AI/ML) at AWS Sydney specialised in machine studying and Amazon SageMaker. He’s obsessed with serving to clients clear up points associated to machine studying workflows and creating new options for them. Exterior of labor, he enjoys enjoying racquet sports activities and touring.

Raghu Ramesha is a Senior ML Options Architect with the Amazon SageMaker Service workforce. He focuses on serving to clients construct, deploy, and migrate ML manufacturing workloads to SageMaker at scale. He focuses on machine studying, AI, and laptop imaginative and prescient domains, and holds a grasp’s diploma in Pc Science from UT Dallas. In his free time, he enjoys touring and pictures.

Shiva Raaj Kotini works as a Principal Product Supervisor within the Amazon SageMaker inference product portfolio. He focuses on mannequin deployment, efficiency tuning, and optimization in SageMaker for inference.

Mohan Gandhi is a Senior Software program Engineer at AWS. He has been with AWS for the final 10 years and has labored on varied AWS providers like EMR, EFA and RDS. At the moment, he’s centered on bettering the SageMaker Inference Expertise. In his spare time, he enjoys climbing and marathons.