Inference Llama 2 fashions with real-time response streaming utilizing Amazon SageMaker

With the fast adoption of generative AI purposes, there’s a want for these purposes to reply in time to cut back the perceived latency with greater throughput. Basis fashions (FMs) are sometimes pre-trained on huge corpora of knowledge with parameters ranging in scale of thousands and thousands to billions and past. Massive language fashions (LLMs) are a kind of FM that generate textual content as a response of the consumer inference. Inferencing these fashions with various configurations of inference parameters could result in inconsistent latencies. The inconsistency might be due to the various variety of response tokens you expect from the mannequin or the kind of accelerator the mannequin is deployed on.

In both case, slightly than ready for the complete response, you possibly can undertake the method of response streaming to your inferences, which sends again chunks of knowledge as quickly as they’re generated. This creates an interactive expertise by permitting you to see partial responses streamed in actual time as a substitute of a delayed full response.

With the official announcement that Amazon SageMaker real-time inference now supports response streaming, now you can repeatedly stream inference responses again to the shopper when utilizing Amazon SageMaker real-time inference with response streaming. This resolution will allow you to construct interactive experiences for varied generative AI purposes reminiscent of chatbots, digital assistants, and music turbines. This submit exhibits you learn how to notice sooner response instances within the type of Time to First Byte (TTFB) and cut back the general perceived latency whereas inferencing Llama 2 fashions.

To implement the answer, we use SageMaker, a totally managed service to organize knowledge and construct, practice, and deploy machine studying (ML) fashions for any use case with totally managed infrastructure, instruments, and workflows. For extra details about the assorted deployment choices SageMaker gives, consult with Amazon SageMaker Model Hosting FAQs. Let’s perceive how we are able to deal with the latency points utilizing real-time inference with response streaming.

Resolution overview

As a result of we wish to deal with the aforementioned latencies related to real-time inference with LLMs, let’s first perceive how we are able to use the response streaming help for real-time inferencing for Llama 2. Nonetheless, any LLM can benefit from response streaming help with real-time inferencing.

Llama 2 is a group of pre-trained and fine-tuned generative textual content fashions ranging in scale from 7 billion to 70 billion parameters. Llama 2 fashions are autoregressive fashions with decoder solely structure. When supplied with a immediate and inference parameters, Llama 2 fashions are able to producing textual content responses. These fashions can be utilized for translation, summarization, query answering, and chat.

For this submit, we deploy the Llama 2 Chat mannequin meta-llama/Llama-2-13b-chat-hf on SageMaker for real-time inferencing with response streaming.

Relating to deploying fashions on SageMaker endpoints, you possibly can containerize the fashions utilizing specialised AWS Deep Learning Container (DLC) photographs out there for well-liked open supply libraries. Llama 2 fashions are textual content technology fashions; you should use both the Hugging Face LLM inference containers on SageMaker powered by Hugging Face Text Generation Inference (TGI) or AWS DLCs for Large Model Inference (LMI).

On this submit, we deploy the Llama 2 13B Chat mannequin utilizing DLCs on SageMaker Internet hosting for real-time inference powered by G5 situations. G5 situations are a high-performance GPU-based situations for graphics-intensive purposes and ML inference. You too can use supported occasion varieties p4d, p3, g5, and g4dn with acceptable modifications as per the occasion configuration.

Stipulations

To implement this resolution, it is best to have the next:

• An AWS account with an AWS Identity and Access Management (IAM) position with permissions to handle assets created as a part of the answer.
• If that is your first time working with Amazon SageMaker Studio, you first must create a SageMaker domain.
• A Hugging Face account. Sign up along with your electronic mail if you happen to don’t have already got account.
  • For seamless entry of the fashions out there on Hugging Face, particularly gated fashions reminiscent of Llama, for fine-tuning and inferencing functions, it is best to have a Hugging Face account to acquire a learn entry token. After you join your Hugging Face account, log in to go to https://huggingface.co/settings/tokens to create a learn entry token.
• Entry to Llama 2, utilizing the identical electronic mail ID that you just used to enroll in Hugging Face.
  • The Llama 2 fashions out there by way of Hugging Face are gated fashions. Using the Llama mannequin is ruled by the Meta license. To obtain the mannequin weights and tokenizer, request access to Llama and settle for their license.
  • After you’re granted entry (sometimes in a few days), you’ll obtain an electronic mail affirmation. For this instance, we use the mannequin Llama-2-13b-chat-hf, however it is best to be capable of entry different variants as properly.

Method 1: Hugging Face TGI

On this part, we present you learn how to deploy the meta-llama/Llama-2-13b-chat-hf mannequin to a SageMaker real-time endpoint with response streaming utilizing Hugging Face TGI. The next desk outlines the specs for this deployment.

Deploy the mannequin

First, you retrieve the bottom picture for the LLM to be deployed. You then construct the mannequin on the bottom picture. Lastly, you deploy the mannequin to the ML occasion for SageMaker Internet hosting for real-time inference.

Let’s observe learn how to obtain the deployment programmatically. For brevity, solely the code that helps with the deployment steps is mentioned on this part. The total supply code for deployment is offered within the pocket book llama-2-hf-tgi/llama-2-13b-chat-hf/1-deploy-llama-2-13b-chat-hf-tgi-sagemaker.ipynb.

Retrieve the newest Hugging Face LLM DLC powered by TGI by way of pre-built SageMaker DLCs. You utilize this picture to deploy the meta-llama/Llama-2-13b-chat-hf mannequin on SageMaker. See the next code:

from sagemaker.huggingface import get_huggingface_llm_image_uri

# retrieve the llm picture uri
llm_image = get_huggingface_llm_image_uri(
  "huggingface",
  model="1.0.3"
)

Outline the setting for the mannequin with the configuration parameters outlined as follows:

instance_type = "ml.g5.12xlarge"
number_of_gpu = 4
config = {
    'HF_MODEL_ID': "meta-llama/Llama-2-13b-chat-hf", # model_id from hf.co/fashions
    'SM_NUM_GPUS': json.dumps(number_of_gpu), # Variety of GPU used per reproduction
    'MAX_INPUT_LENGTH': json.dumps(2048),  # Max size of enter textual content
    'MAX_TOTAL_TOKENS': json.dumps(4096),  # Max size of the technology (together with enter textual content)
    'MAX_BATCH_TOTAL_TOKENS': json.dumps(8192),  # Limits the variety of tokens that may be processed in parallel in the course of the technology
    'HUGGING_FACE_HUB_TOKEN': "<YOUR_HUGGING_FACE_READ_ACCESS_TOKEN>"
}

Substitute <YOUR_HUGGING_FACE_READ_ACCESS_TOKEN> for the config parameter HUGGING_FACE_HUB_TOKEN with the worth of the token obtained out of your Hugging Face profile as detailed within the stipulations part of this submit. Within the configuration, you outline the variety of GPUs used per reproduction of a mannequin as 4 for SM_NUM_GPUS. Then you possibly can deploy the meta-llama/Llama-2-13b-chat-hf mannequin on an ml.g5.12xlarge occasion that comes with 4 GPUs.

Now you possibly can construct the occasion of HuggingFaceModel with the aforementioned setting configuration:

llm_model = HuggingFaceModel(
    position=position,
    image_uri=llm_image,
    env=config
)

Lastly, deploy the mannequin by offering arguments to the deploy technique out there on the mannequin with varied parameter values reminiscent of endpoint_name, initial_instance_count, and instance_type:

llm = llm_model.deploy(
    endpoint_name=endpoint_name,
    initial_instance_count=1,
    instance_type=instance_type,
    container_startup_health_check_timeout=health_check_timeout,
)

Carry out inference

The Hugging Face TGI DLC comes with the flexibility to stream responses with none customizations or code modifications to the mannequin. You should use invoke_endpoint_with_response_stream in case you are utilizing Boto3 or InvokeEndpointWithResponseStream when programming with the SageMaker Python SDK.

The InvokeEndpointWithResponseStream API of SageMaker permits builders to stream responses again from SageMaker fashions, which will help enhance buyer satisfaction by decreasing the perceived latency. That is particularly essential for purposes constructed with generative AI fashions, the place quick processing is extra essential than ready for your entire response.

For this instance, we use Boto3 to deduce the mannequin and use the SageMaker API invoke_endpoint_with_response_stream as follows:

def get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload):
    response_stream = sagemaker_runtime.invoke_endpoint_with_response_stream(
        EndpointName=endpoint_name,
        Physique=json.dumps(payload), 
        ContentType="software/json",
        CustomAttributes="accept_eula=false"
    )
    return response_stream

The argument CustomAttributes is about to the worth accept_eula=false. The accept_eula parameter have to be set to true to efficiently get hold of the response from the Llama 2 fashions. After the profitable invocation utilizing invoke_endpoint_with_response_stream, the strategy will return a response stream of bytes.

The next diagram illustrates this workflow.

You want an iterator that loops over the stream of bytes and parses them to readable textual content. The LineIterator implementation may be discovered at llama-2-hf-tgi/llama-2-13b-chat-hf/utils/LineIterator.py. Now you’re prepared to organize the immediate and directions to make use of them as a payload whereas inferencing the mannequin.

Put together a immediate and directions

On this step, you put together the immediate and directions to your LLM. To immediate Llama 2, it is best to have the next immediate template:

<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>

{{ user_message }} [/INST]

You construct the immediate template programmatically outlined within the technique build_llama2_prompt, which aligns with the aforementioned immediate template. You then outline the directions as per the use case. On this case, we’re instructing the mannequin to generate an electronic mail for a advertising and marketing marketing campaign as coated within the get_instructions technique. The code for these strategies is within the llama-2-hf-tgi/llama-2-13b-chat-hf/2-sagemaker-realtime-inference-llama-2-13b-chat-hf-tgi-streaming-response.ipynb pocket book. Construct the instruction mixed with the duty to be carried out as detailed in user_ask_1 as follows:

user_ask_1 = f'''
AnyCompany just lately introduced new service launch named AnyCloud Web Service.
Write a brief electronic mail concerning the product launch with Name to motion to Alice Smith, whose electronic mail is alice.smith@instance.com
Point out the Coupon Code: EARLYB1RD to get 20% for 1st 3 months.
'''
directions = get_instructions(user_ask_1)
immediate = build_llama2_prompt(directions)

We move the directions to construct the immediate as per the immediate template generated by build_llama2_prompt.

inference_params = {
        "do_sample": True,
        "top_p": 0.6,
        "temperature": 0.9,
        "top_k": 50,
        "max_new_tokens": 512,
        "repetition_penalty": 1.03,
        "cease": ["</s>"],
        "return_full_text": False
    }
payload = {
    "inputs":  immediate,
    "parameters": inference_params,
    "stream": True ## <-- to have response stream.
}

We membership the inference parameters together with immediate with the important thing stream with the worth True to type a remaining payload. Ship the payload to get_realtime_response_stream, which might be used to invoke an endpoint with response streaming:

resp = get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload)
print_response_stream(resp)

The generated textual content from the LLM might be streamed to the output as proven within the following animation.

Method 2: LMI with DJL Serving

On this part, we display learn how to deploy the meta-llama/Llama-2-13b-chat-hf mannequin to a SageMaker real-time endpoint with response streaming utilizing LMI with DJL Serving. The next desk outlines the specs for this deployment.

You first obtain the mannequin and retailer it in Amazon Simple Storage Service (Amazon S3). You then specify the S3 URI indicating the S3 prefix of the mannequin within the serving.properties file. Subsequent, you retrieve the bottom picture for the LLM to be deployed. You then construct the mannequin on the bottom picture. Lastly, you deploy the mannequin to the ML occasion for SageMaker Internet hosting for real-time inference.

Let’s observe learn how to obtain the aforementioned deployment steps programmatically. For brevity, solely the code that helps with the deployment steps is detailed on this part. The total supply code for this deployment is offered within the pocket book llama-2-lmi/llama-2-13b-chat/1-deploy-llama-2-13b-chat-lmi-response-streaming.ipynb.

Obtain the mannequin snapshot from Hugging Face and add the mannequin artifacts on Amazon S3

With the aforementioned stipulations, obtain the mannequin on the SageMaker pocket book occasion after which add it to the S3 bucket for additional deployment:

model_name="meta-llama/Llama-2-13b-chat-hf"
# Solely obtain pytorch checkpoint information
allow_patterns = ["*.json", "*.txt", "*.model", "*.safetensors", "*.bin", "*.chk", "*.pth"]

# Obtain the mannequin snapshot
model_download_path = snapshot_download(
    repo_id=model_name, 
    cache_dir=local_model_path, 
    allow_patterns=allow_patterns, 
    token='<YOUR_HUGGING_FACE_READ_ACCESS_TOKEN>'
)

Notice that though you don’t present a legitimate entry token, the mannequin will obtain. However once you deploy such a mannequin, the mannequin serving gained’t succeed. Subsequently, it’s really helpful to exchange <YOUR_HUGGING_FACE_READ_ACCESS_TOKEN> for the argument token with the worth of the token obtained out of your Hugging Face profile as detailed within the stipulations. For this submit, we specify the official mannequin’s title for Llama 2 as recognized on Hugging Face with the worth meta-llama/Llama-2-13b-chat-hf. The uncompressed mannequin might be downloaded to local_model_path because of operating the aforementioned code.

Add the information to Amazon S3 and procure the URI, which might be later utilized in serving.properties.

You may be packaging the meta-llama/Llama-2-13b-chat-hf mannequin on the LMI container picture with DJL Serving utilizing the configuration specified by way of serving.properties. You then deploy the mannequin together with mannequin artifacts packaged on the container picture on the SageMaker ML occasion ml.g5.12xlarge. You then use this ML occasion for SageMaker Internet hosting for real-time inferencing.

Put together mannequin artifacts for DJL Serving

Put together your mannequin artifacts by making a serving.properties configuration file:

%%writefile chat_llama2_13b_hf/serving.properties
engine = MPI
choice.entryPoint=djl_python.huggingface
choice.tensor_parallel_degree=4
choice.low_cpu_mem_usage=TRUE
choice.rolling_batch=lmi-dist
choice.max_rolling_batch_size=64
choice.model_loading_timeout=900
choice.model_id={{model_id}}
choice.paged_attention=true

We use the next settings on this configuration file:

• engine – This specifies the runtime engine for DJL to make use of. The attainable values embrace Python, DeepSpeed, FasterTransformer, and MPI. On this case, we set it to MPI. Mannequin Parallelization and Inference (MPI) facilitates partitioning the mannequin throughout all of the out there GPUs and subsequently accelerates inference.
• choice.entryPoint – This feature specifies which handler supplied by DJL Serving you want to use. The attainable values are djl_python.huggingface, djl_python.deepspeed, and djl_python.stable-diffusion. We use djl_python.huggingface for Hugging Face Speed up.
• choice.tensor_parallel_degree – This feature specifies the variety of tensor parallel partitions carried out on the mannequin. You’ll be able to set to the variety of GPU gadgets over which Speed up must partition the mannequin. This parameter additionally controls the variety of staff per mannequin that might be began up when DJL serving runs. For instance, if we have now a 4 GPU machine and we’re creating 4 partitions, then we may have one employee per mannequin to serve the requests.
• choice.low_cpu_mem_usage – This reduces CPU reminiscence utilization when loading fashions. We advocate that you just set this to TRUE.
• choice.rolling_batch – This allows iteration-level batching utilizing one of many supported methods. Values embrace auto, scheduler, and lmi-dist. We use lmi-dist for turning on steady batching for Llama 2.
• choice.max_rolling_batch_size – This limits the variety of concurrent requests within the steady batch. The worth defaults to 32.
• choice.model_id – It’s best to exchange {{model_id}} with the mannequin ID of a pre-trained mannequin hosted inside a model repository on Hugging Face or S3 path to the mannequin artifacts.

Extra configuration choices may be present in Configurations and settings.

As a result of DJL Serving expects the mannequin artifacts to be packaged and formatted in a .tar file, run the next code snippet to compress and add the .tar file to Amazon S3:

s3_code_prefix = f"{s3_prefix}/code" # folder inside bucket the place code artifact will go
s3_code_artifact = sess.upload_data("mannequin.tar.gz", bucket, s3_code_prefix)

Retrieve the newest LMI container picture with DJL Serving

Subsequent, you employ the DLCs out there with SageMaker for LMI to deploy the mannequin. Retrieve the SageMaker picture URI for the djl-deepspeed container programmatically utilizing the next code:

from sagemaker import image_uris
inference_image_uri = image_uris.retrieve(
    framework="djl-deepspeed", area=area, model="0.25.0"
)

You should use the aforementioned picture to deploy the meta-llama/Llama-2-13b-chat-hf mannequin on SageMaker. Now you possibly can proceed to create the mannequin.

Create the mannequin

You’ll be able to create the mannequin whose container is constructed utilizing the inference_image_uri and the mannequin serving code situated on the S3 URI indicated by s3_code_artifact:

from sagemaker.utils import name_from_base

model_name = name_from_base(f"Llama-2-13b-chat-lmi-streaming")

create_model_response = sm_client.create_model(
    ModelName=model_name,
    ExecutionRoleArn=position,
    PrimaryContainer={
        "Picture": inference_image_uri,
        "ModelDataUrl": s3_code_artifact,
        "Surroundings": {"MODEL_LOADING_TIMEOUT": "3600"},
    },
)

Now you possibly can create the mannequin config with all the main points for the endpoint configuration.

Create the mannequin config

Use the next code to create a mannequin config for the mannequin recognized by model_name:

endpoint_config_name = f"{model_name}-config"

endpoint_name = name_from_base(model_name)

endpoint_config_response = sm_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            "VariantName": "variant1",
            "ModelName": model_name,
            "InstanceType": "ml.g5.12xlarge",
            "InitialInstanceCount": 1,
            "ModelDataDownloadTimeoutInSeconds": 3600,
            "ContainerStartupHealthCheckTimeoutInSeconds": 3600,
        },
    ],
)

The mannequin config is outlined for the ProductionVariants parameter InstanceType for the ML occasion ml.g5.12xlarge. You additionally present the ModelName utilizing the identical title that you just used to create the mannequin within the earlier step, thereby establishing a relation between the mannequin and endpoint configuration.

Now that you’ve got outlined the mannequin and mannequin config, you possibly can create the SageMaker endpoint.

Create the SageMaker endpoint

Create the endpoint to deploy the mannequin utilizing the next code snippet:

create_endpoint_response = sm_client.create_endpoint(
    EndpointName=f"{endpoint_name}", EndpointConfigName=endpoint_config_name
)

You’ll be able to view the progress of the deployment utilizing the next code snippet:

resp = sm_client.describe_endpoint(EndpointName=endpoint_name)
standing = resp["EndpointStatus"]

After the deployment is profitable, the endpoint standing might be InService. Now that the endpoint is prepared, let’s carry out inference with response streaming.

Actual-time inference with response streaming

As we coated within the earlier method for Hugging Face TGI, you should use the identical technique get_realtime_response_stream to invoke response streaming from the SageMaker endpoint. The code for inferencing utilizing the LMI method is within the llama-2-lmi/llama-2-13b-chat/2-inference-llama-2-13b-chat-lmi-response-streaming.ipynb pocket book. The LineIterator implementation is situated in llama-2-lmi/utils/LineIterator.py. Notice that the LineIterator for the Llama 2 Chat mannequin deployed on the LMI container is completely different to the LineIterator referenced in Hugging Face TGI part. The LineIterator loops over the byte stream from Llama 2 Chat fashions inferenced with the LMI container with djl-deepspeed model 0.25.0. The next helper perform will parse the response stream acquired from the inference request made by way of the invoke_endpoint_with_response_stream API:

from utils.LineIterator import LineIterator

def print_response_stream(response_stream):
    event_stream = response_stream.get('Physique')
    for line in LineIterator(event_stream):
        print(line, finish='')

The previous technique prints the stream of knowledge learn by the LineIterator in a human-readable format.

Let’s discover learn how to put together the immediate and directions to make use of them as a payload whereas inferencing the mannequin.

Since you’re inferencing the identical mannequin in each Hugging Face TGI and LMI, the method of making ready the immediate and directions is similar. Subsequently, you should use the strategies get_instructions and build_llama2_prompt for inferencing.

The get_instructions technique returns the directions. Construct the directions mixed with the duty to be carried out as detailed in user_ask_2 as follows:

user_ask_2 = f'''
AnyCompany just lately introduced new service launch named AnyCloud Streaming Service.
Write a brief electronic mail concerning the product launch with Name to motion to Alice Smith, whose electronic mail is alice.smith@instance.com
Point out the Coupon Code: STREAM2DREAM to get 15% for 1st 6 months.
'''

directions = get_instructions(user_ask_2)
immediate = build_llama2_prompt(directions)

Go the directions to construct the immediate as per the immediate template generated by build_llama2_prompt:

inference_params = {
        "do_sample": True,
        "top_p": 0.6,
        "temperature": 0.9,
        "top_k": 50,
        "max_new_tokens": 512,
        "return_full_text": False,
    }

payload = {
    "inputs":  immediate,
    "parameters": inference_params
}

We membership the inference parameters together with the immediate to type a remaining payload. You then ship the payload to get_realtime_response_stream, which is used to invoke an endpoint with response streaming:

resp = get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload)
print_response_stream(resp)

The generated textual content from the LLM might be streamed to the output as proven within the following animation.

Clear up

To keep away from incurring pointless prices, use the AWS Management Console to delete the endpoints and its related assets that had been created whereas operating the approaches talked about within the submit. For each deployment approaches, carry out the next cleanup routine:

import boto3
sm_client = boto3.shopper('sagemaker')
endpoint_name="<SageMaker_Real-time_Endpoint_Name>"
endpoint = sm_client.describe_endpoint(EndpointName=endpoint_name)
endpoint_config_name = endpoint['EndpointConfigName']
endpoint_config = sm_client.describe_endpoint_config(EndpointConfigName=endpoint_config_name)
model_name = endpoint_config['ProductionVariants'][0]['ModelName']

print(f"""
About to delete the next sagemaker assets:
Endpoint: {endpoint_name}
Endpoint Config: {endpoint_config_name}
Mannequin: {model_name}
""")

# delete endpoint
sm_client.delete_endpoint(EndpointName=endpoint_name)
# delete endpoint config
sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)
# delete mannequin
sm_client.delete_model(ModelName=model_name)

Substitute <SageMaker_Real-time_Endpoint_Name> for variable endpoint_name with the precise endpoint.

For the second method, we saved the mannequin and code artifacts on Amazon S3. You’ll be able to clear up the S3 bucket utilizing the next code:

s3 = boto3.useful resource('s3')
s3_bucket = s3.Bucket(bucket)
s3_bucket.objects.filter(Prefix=s3_prefix).delete()

Conclusion

On this submit, we mentioned how a various variety of response tokens or a special set of inference parameters can have an effect on the latencies related to LLMs. We confirmed learn how to deal with the issue with the assistance of response streaming. We then recognized two approaches for deploying and inferencing Llama 2 Chat fashions utilizing AWS DLCs—LMI and Hugging Face TGI.

It’s best to now perceive the significance of streaming response and the way it can cut back perceived latency. Streaming response can enhance the consumer expertise, which in any other case would make you wait till the LLM builds the entire response. Moreover, deploying Llama 2 Chat fashions with response streaming improves the consumer expertise and makes your prospects completely happy.

You’ll be able to consult with the official aws-samples amazon-sagemaker-llama2-response-streaming-recipes that covers deployment for different Llama 2 mannequin variants.

References

Concerning the Authors

Pavan Kumar Rao Navule is a Options Architect at Amazon Internet Companies. He works with ISVs in India to assist them innovate on AWS. He’s a broadcast writer for the guide “Getting Began with V Programming.” He pursued an Govt M.Tech in Information Science from the Indian Institute of Expertise (IIT), Hyderabad. He additionally pursued an Govt MBA in IT specialization from the Indian Faculty of Enterprise Administration and Administration, and holds a B.Tech in Electronics and Communication Engineering from the Vaagdevi Institute of Expertise and Science. Pavan is an AWS Licensed Options Architect Skilled and holds different certifications reminiscent of AWS Licensed Machine Studying Specialty, Microsoft Licensed Skilled (MCP), and Microsoft Licensed Expertise Specialist (MCTS). He’s additionally an open-source fanatic. In his free time, he likes to hearken to the nice magical voices of Sia and Rihanna.

Sudhanshu Hate is principal AI/ML specialist with AWS and works with shoppers to advise them on their MLOps and generative AI journey. In his earlier position earlier than Amazon, he conceptualized, created, and led groups to construct ground-up open source-based AI and gamification platforms, and efficiently commercialized it with over 100 shoppers. Sudhanshu to his credit score a few patents, has written two books and several other papers and blogs, and has introduced his factors of view in varied technical boards. He has been a thought chief and speaker, and has been within the trade for almost 25 years. He has labored with Fortune 1000 shoppers throughout the globe and most just lately with digital native shoppers in India.