Deploying Giant NLP Fashions: Infrastructure Value Optimization

NLP fashions in business purposes akin to textual content era techniques have skilled nice curiosity among the many consumer. These fashions have achieved numerous groundbreaking ends in many NLP duties like question-answering, summarization, language translation, classification, paraphrasing, et cetera. 

Fashions like for instance ChatGPT, Gopher **(280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) are predominantly very massive and sometimes addressed as massive language fashions or LLMs. These fashions can simply have tens of millions or as much as billions of parameters making them financially costly to deploy and keep.

Such massive natural language processing models require important computational energy and reminiscence, which is usually the main reason behind excessive infrastructure prices. Even if you’re fine-tuning an average-sized mannequin for a large-scale utility, you want to muster an enormous quantity of knowledge. 

Such situations inevitably result in stacking new layers of neural connections, making it a big mannequin, furthermore, deploying these fashions would require quick and costly GPU, which can finally add to the infrastructure price. So is there a solution to maintain these bills in verify?

Certain there may be.

This text goals to offer some methods, suggestions, and methods you may apply to optimize your infrastructure whereas deploying them. Within the following sections, we are going to discover these:

Challenges of enormous NLP fashions

Computational assets

LLMs require is a major quantity of assets for optimum efficiency. Beneath are the challenges which might be often confronted regarding the identical. 

1. Excessive computational necessities

Deploying LLMs could be difficult as they require important computational assets to carry out inference. That is very true when the mannequin is used for real-time purposes, akin to chatbots or digital assistants. 

Take into account ChatGPT for instance. It’s able to processing and responding to queries immediately inside seconds (more often than not). However there are occasions when the consumer site visitors appears to be greater, throughout these moments, the inference time will get greater. There are different elements that may delay the inference, such because the complexity of the query, the quantity of knowledge required to generate a response, et cetera. However in any case, if the mannequin is meant to serve in real-time, it should be able to excessive throughput and low latency.

2. Storage capability

With parameters starting from tens of millions to billions, LLM can pose storage capability challenges. It is going to be good to retailer the entire mannequin in a single storage system, however due to the dimensions, it isn’t doable. 

For instance, OpenAI’s GPT-3 mannequin, with 175B parameters, requires over 300GB of storage for its parameters alone. Moreover, it requires a GPU with a minimal of 16GB of reminiscence to run effectively. Storing and working such a big mannequin on a single system could also be impractical for a lot of use circumstances as a result of {hardware} necessities. As such, there are three principal points round storage capability with LLMs:

2.1 Reminiscence limitations

LLMs require a variety of reminiscence as they course of an enormous quantity of knowledge. This may be difficult, particularly if you wish to deploy them on a low-memory system akin to a cell phone. 

One solution to deploy such fashions is to make use of a distributed system or distributed inference. In distributed inference, the mannequin is distributed on a number of nodes or servers. It permits the distribution of the workload and quickens the method. However the problem right here is that it might require important experience to arrange and keep. Plus, the bigger the mannequin, the extra servers are required, which once more will increase the deployment price. 

2.2 Giant mannequin sizes

The MT-NLG mannequin launched in 2022 has 530 billion parameters and requires a number of hundred gigabytes of storage. Excessive-end GPUs and primary knowledge parallelism aren’t enough for deployment, and even different options like pipeline and mannequin parallelism have trade-offs between performance, usability, and reminiscence/compute effectivity. Because the authors within the paper “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models put it, this, in flip, reduces the effectiveness of the mannequin. 

For example, a 1.5B parameter mannequin on 32GB can simply run out of reminiscence throughout inference if the enter question is lengthy and complex. Even for primary inference on LLM, a number of accelerators or multi-node computing clusters like a number of Kubernetes pods are required. There are methods mentioned by researchers the place they suggest the concept of offloading parameters to the native RAM. However these methods turned out to be inefficient in sensible use-case situations. Customers can’t obtain such massive scaled fashions on their techniques simply to translate or summarise a given textual content. 

2.3 Scalability challenges 

One other space for enchancment with LLMs is scalability. We all know that a big mannequin is usually scaled utilizing mannequin parallelism (MP), which requires a number of storage and reminiscence capability. This entails dividing the mannequin into smaller components and distributing it throughout a number of machines. Every machine processes a distinct a part of the mannequin, and the outcomes are mixed to provide the ultimate output. This method could be useful in dealing with massive fashions, however it requires cautious consideration of the communication overhead between machines. 

In Distributed inference, LLM is deployed on a number of machines, with every machine processing a subset of the enter knowledge. This strategy is crucial for dealing with large-scale language duties that require enter to go by billions of parameters. 

More often than not, MP works, however there are situations the place it doesn’t. The reason is MP divides the mannequin vertically, distributing the computation and parameters amongst a number of units for every layer the place the inter-GPU communication bandwidth is massive. This distribution facilitates intensive communication between every layer in a single node. The limitation comes outdoors a single node which primarily results in a fall in efficiency and effectivity.

3. Bandwidth necessities

As mentioned beforehand, LLM must be scaled utilizing MP. However the situation we discovered was that MP is environment friendly in single-node clusters, however in a multi-node setting, the inference isn’t environment friendly. That is due to the low bandwidth networks. 

Deploying a big language mannequin requires a number of community requests to retrieve knowledge from totally different servers. Community latency can influence the time required to switch knowledge between the servers, which may end up in slower efficiency, ultimately resulting in excessive latency and response time. This will trigger delays in processing, which may influence consumer expertise.

4. Useful resource constraints

Restricted storage capability can limit the flexibility to retailer a number of variations of the identical mannequin, which may make it troublesome to check the efficiency of various fashions and observe the progress of mannequin improvement over time. This may be true if you wish to undertake a shadow deployment technique.

Power consumption

As mentioned above already, serving LLMs require important computational assets, which may result in excessive power consumption and a big carbon footprint. This may be problematic for organizations which might be dedicated to lowering their environmental influence.

Only for reference, beneath is the picture exhibiting the monetary estimation of the LLMs, together with the carbon footprint that they produce throughout coaching.

What’s extra stunning is that 80-90% of the machine studying workload is inference processing, in line with NVIDIA. Likewise, in line with AWS, inference accounts for 90% of machine studying demand within the cloud.

Value

Deploying and utilizing LLMs could be expensive, together with the price of {hardware}, storage, and infrastructure. Moreover, the price of deploying the mannequin could be important, particularly when utilizing assets akin to GPUs or TPUs for low latency and excessive throughput throughout inference. This will make it difficult for smaller organizations or people to make use of LLMs for his or her purposes.

To place this into perspective, it’s anticipated that the working price of the chatGPT is round $100,000 per day or $3M monthly.

Methods for optimizing infrastructure prices of enormous NLP fashions

On this part, we are going to discover and focus on the doable options and methods for the challenges mentioned within the earlier part. It’s value noting that if you deploy the mannequin on the cloud, you select the inference possibility and thereby create an end-point. See the picture beneath. 

Hold that in thoughts, and with all of the challenges we mentioned earlier, we are going to focus on methods that can be utilized to optimize the fee round this infrastructure for deploying LLMs. Beneath are among the steps that you could comply with to deploy your mannequin as effectively as doable. 

Good use of cloud computing for computational assets

Utilizing cloud computing providers can present on-demand entry to highly effective computing assets, together with CPUs and GPUs. Cloud computing providers are versatile and may scale in line with your necessities. 

One of many necessary suggestions is that you need to make a price range in your mission. Making a price range at all times helps you discover methods to optimize your mission that won’t exceed your monetary limitation. 

Now with regards to cloud providers, there are a variety of firms that supply their platform. Cloud suppliers akin to Amazon Internet Companies (AWS), Microsoft Azure, and Google Cloud Platform supply a variety of choices for deploying LLMs, together with digital machines, containers, and serverless computing. However regardless of you should do your individual analysis and calculation. For example, you should know these three issues:

After you have the small print, you may really calculate how much-accelerated computing energy you want. Based mostly upon that, you may plan and execute your mannequin deployment. 

Calculating mannequin dimension

You’ll be able to see the desk beneath, which provides you with an concept of what number of FLOPs you may want in your mannequin. After you have an estimation, you may then go forward and discover the related GPU in your most well-liked cloud platform. 

A software that I discovered beneath the weblog put up named “Estimating Training Compute of Deep Learning Models”  lets you calculate the FLOPs required in your mannequin each for coaching and inference. 

The app relies on the works of Kaplan et al., 2020 or Hoffman et al., 2022 the place they present how you can practice a mannequin on a fixed-compute price range. To know extra on this topic you may learn the weblog here.

Deciding on the appropriate {hardware}

After you have calculated the required FLOPs, you may go forward and select the GPU. Be sure to are conscious of the options that the GPU presents. For example, see the picture beneath to get an understanding.

Above you may see the record of specs that NVIDIA presents. Equally, you may examine totally different GPUs and see which one fits your price range. 

Choosing the proper inference possibility

After you have calculated the mannequin dimension and chosen the GPU, you may then proceed to decide on the inference possibility. Amazon SageMaker presents a number of inference choices to swimsuit totally different workloads. For example, in case you require:

1. Actual-time inference, which is appropriate for low-latency or high-throughput on-line inferences and helps payload sizes as much as 6 MB and processing occasions of 60 seconds.
2. Serverless inference, which is good for intermittent or unpredictable site visitors patterns and helps payload sizes as much as 4 MB and processing occasions of 60 seconds. In serverless inference, the mannequin scales robotically based mostly on the incoming site visitors or requests. At occasions when the mannequin is sitting idle you received’t be charged. It presents a pay-as-you-use facility. 
3. Batch remodel is appropriate for offline processing of enormous datasets and helps payload sizes of GBs and processing occasions of days. 
4. Asynchronous inference is appropriate for queuing requests with massive payloads and lengthy processing occasions, helps payloads as much as 1 GB and processing occasions as much as one hour, and may scale all the way down to 0 when there are not any requests.

To get a greater understanding and meet your requirement, have a look at the picture beneath. 

When all of the above factors are glad, you may then deploy the mannequin on any of the cloud providers. 

To shortly summarize:

Optimizing the mannequin for serving

Within the final part, I mentioned how the dimensions of LLMs can pose an issue for deployment. When your mannequin is just too massive, methods like mannequin compilation, mannequin compression, and mannequin sharding can be utilized. These methods cut back the dimensions of the mannequin whereas preserving accuracy, which permits simpler deployment and cut back the related bills considerably. 

Let’s discover every of these intimately. 

Mannequin compression

Mannequin compression is a method used to optimize and remodel an LLM into an environment friendly executable mannequin that may be run on specialised {hardware} or software program platforms–often cloud providers. The objective of mannequin compression is to enhance the efficiency and effectivity of LLM inference by leveraging hardware-specific optimizations, akin to lowered reminiscence footprint, improved computation parallelism, and lowered latency.

It is a good approach as a result of it lets you play with a distinct mixture, set efficiency benchmarks for numerous duties, and discover a value that fits your price range.  As such, mannequin compression entails a number of steps:

1. Graph optimization: The high-level LLM graph is remodeled and optimized utilizing graph optimization methods akin to pruning and quantization to scale back the computational complexity and reminiscence footprint of the mannequin. This, in flip, makes the mannequin small whereas preserving its accuracy. 
2. {Hardware}-specific optimization: The optimized LLM graph is additional optimized to leverage hardware-specific optimizations. For example, Amazon Sagemaker offers mannequin serving containers for numerous standard ML frameworks, together with XGBoost, scikit-learn, PyTorch, TensorFlow, and Apache MXNet, together with software program improvement kits (SDKs) for every container.

Listed below are a number of mannequin compression methods that one should know.

Mannequin quantization

Mannequin quantization (MQ) is a method used to scale back the reminiscence footprint and computation necessities of an LLM. MQ primarily transforms the mannequin parameters and activations with lower-precision knowledge sorts. The objective of mannequin quantization is to enhance the effectivity of LLM throughout inference by lowering the reminiscence bandwidth necessities and exploiting hardware-specific optimizations optimized for lower-precision arithmetic.

PyTorch presents mannequin quantization, their API entails the discount of mannequin parameters by an element of 4, whereas the reminiscence bandwidth required by the mannequin by the issue 2 to 4 occasions. Because of these enhancements, the inference pace can enhance by 2 to 4 occasions, owing to the discount in reminiscence bandwidth necessities and quicker computations utilizing int8 arithmetic. Nonetheless, the exact diploma of acceleration achieved depends upon the {hardware}, runtime, and mannequin used.

There are a number of approaches to mannequin quantization for LLMs, together with:

Mannequin quantization could be difficult to implement successfully, because it requires cautious consideration of the trade-offs between lowered precision and mannequin accuracy, in addition to the hardware-specific optimizations that may be leveraged with lower-precision arithmetic. Nonetheless, when finished appropriately, mannequin quantization can considerably enhance the effectivity of LLM inference, enabling higher real-time inference on large-scale datasets and edge units.

1. Put up-training quantization: On this strategy, the LLM is first educated utilizing floating-point knowledge sorts, after which the weights and activations are quantized to lower-precision knowledge sorts post-training. This strategy is straightforward to implement and may obtain good accuracy with a cautious collection of quantization parameters.
2. Quantization-aware coaching: Right here, the LLM is quantized throughout coaching, permitting the mannequin to adapt to the lowered precision throughout coaching. This strategy can obtain greater accuracy than post-training quantization however requires extra computation throughout coaching.
3. Hybrid quantization: It combines each post-training quantization and quantization-aware coaching, permitting the LLM to adapt to lower-precision knowledge sorts throughout coaching whereas additionally making use of post-training quantization to additional cut back the reminiscence footprint and computational complexity of the mannequin.

Mannequin Pruning

Mannequin pruning (MP) is once more a method used to scale back the dimensions and computational complexity of an LLM by eradicating redundant or pointless mannequin parameters. MP is to enhance the effectivity of LLM inference with out sacrificing accuracy.

MP entails figuring out and eradicating redundant or pointless mannequin parameters utilizing numerous pruning algorithms. These algorithms could be broadly categorized into two classes:

1. Weight pruning: In weight pruning, particular person weights within the LLM are eliminated based mostly on their magnitude or significance, utilizing methods akin to magnitude-based pruning or structured pruning. Weight pruning can considerably cut back the variety of mannequin parameters and the computational complexity of the LLM, however it might require fine-tuning of the pruned mannequin to keep up its accuracy.
2. Neuron pruning: In neuron pruning, complete neurons or activations within the LLM are eliminated based mostly on their significance, utilizing methods akin to channel pruning or neuron-level pruning. Neuron pruning may considerably cut back the variety of mannequin parameters and the computational complexity of the LLM, however it might be harder to implement and should require extra in depth retraining and possibly fine-tuning to keep up accuracy.

Listed below are a few approaches to mannequin pruning:

1. Put up-training pruning: On this strategy, the LLM is first educated utilizing customary methods after which pruned utilizing one of many pruning algorithms. The pruned LLM is then fine-tuned to protect its accuracy.
2. Iterative pruning: Right here, the mannequin is educated utilizing customary coaching methods after which pruned iteratively over a number of rounds of coaching and pruning. This strategy can obtain greater ranges of pruning whereas preserving accuracy.

You’ll be able to discover this Colab pocket book by PyTorch to higher perceive MP. 

Mannequin distillation

(MD) is a method used to switch information from an LLM known as a instructor to a smaller, extra environment friendly mannequin known as the scholar. It’s used within the context of mannequin compression. In a nutshell, the instructor mannequin offers steering and suggestions to the scholar mannequin throughout coaching. See the picture beneath.

MD entails coaching a pupil, a extra environment friendly mannequin to imitate the habits of a instructor, extra complicated LLM. The scholar mannequin is ready utilizing a mixture of labeled knowledge and the output possibilities of the bigger LLM. 

There are a number of approaches to mannequin distillation for LLMs, together with:

1. Data distillation: On this strategy, the smaller mannequin is educated to imitate the output possibilities of the bigger LLM utilizing a temperature scaling issue. The temperature scaling issue is used to melt the output possibilities of the instructor mannequin, permitting the smaller mannequin to study from the instructor mannequin’s habits extra successfully.

2. Self-distillation: On this strategy, the bigger LLM is used to generate coaching examples for the smaller mannequin by making use of the instructor mannequin to unlabeled knowledge. The smaller mannequin is then educated on these generated examples, permitting it to study from the habits of the bigger LLM with out requiring labeled knowledge.

3. Ensemble distillation: On this strategy, a number of smaller fashions are educated to imitate the habits of various sub-components of the bigger LLM. The outputs of those smaller fashions are mixed to kind an ensemble mannequin that approximates the habits of the bigger LLM.

Optimizing {hardware} and software program necessities

{Hardware} is a vital space with regards to deploying LLMs. Listed below are some helpful steps you may take for optimizing the {hardware} efficiency:

1. Select {hardware} that matches the LLM’s necessities: Relying on the LLM’s dimension and complexity, you could want {hardware} with a considerable amount of RAM, high-speed storage, or a number of GPUs to hurry up inference. Go for {hardware} that gives the required processing energy, reminiscence, and storage capability, with out overspending on irrelevant options.

2. Use specialised {hardware}: You should utilize specialised {hardware} akin to TPUs (Tensor Processing Models) or FPGAs (Subject-Programmable Gate Arrays) which might be designed particularly for deep studying duties. Equally, accelerated linear algebra or XLA could be leveraged throughout inference time. 

Though such {hardware} could be costly, there are good methods to devour them. You’ll be able to go for charge-on-demand for the {hardware} used. For example, elastic Inference from AWS Sagemaker helps you decrease your price when the mannequin shouldn’t be totally using the GPU occasion for inference. 

3. Use optimized libraries: You should utilize optimized libraries akin to TensorFlow, PyTorch, or JAX  that leverage hardware-specific options to hurry up computation while not having further {hardware}. 

4. Tune the batch dimension: Take into account tuning the batch dimension throughout inference to maximise {hardware} utilization and enhance inference pace. This inherently reduces the {hardware} requirement, thus slicing the fee. 

5. Monitor and optimize: Lastly, monitor the LLM’s efficiency throughout deployment and optimize the {hardware} configuration as wanted to attain the very best efficiency.

Value environment friendly scalability

Right here’s how one can scale your massive NLP fashions whereas preserving prices in verify:

1. Select the appropriate inference possibility, that scales robotically just like the serverless inference possibility. As it’ll cut back the deployment price when the demand is much less. 

A inflexible structure will at all times occupy the identical quantity of reminiscence even when the demand is low thus the deployment and upkeep prices would be the identical. Quite the opposite, a scalable structure can scale horizontally or vertically to accommodate an elevated workload and return to its unique configuration when the mannequin lies in a dormant state. Such an strategy can cut back the price of upkeep each time the extra nodes will not be getting used. 

2. Optimize inference efficiency, by utilizing {hardware} acceleration, akin to GPUs or TPUs, and by optimizing the inference code.

3. Amazon’s Elastic inference is one more nice possibility because it reduces the fee by as much as 75% as a result of the mannequin now not has additional GPUs to compute for inference. For extra on Elastic inference, learn this text here. 

Reducing power prices

1. Select an energy-efficient cloud infrastructure, that makes use of renewable power sources or carbon offsets to scale back the carbon footprint of their knowledge facilities. You too can contemplate selecting energy-efficient GPUs. Try this article by Wired to know extra. 

2. Use caching which helps cut back the computational necessities of LLM inference by storing continuously requested responses in reminiscence. This will considerably cut back the variety of computations required to generate responses to consumer requests. It additionally helps in addressing bandwidth points because it reduces the time to entry knowledge. You’ll be able to retailer continuously accessed knowledge in cache reminiscence in order that it may be shortly accessed with out the necessity for extra bandwidth. This permits you to not go for further storage and reminiscence units. 

Deploying massive NLP fashions: different helpful suggestions

Estimating the NLP mannequin dimension earlier than coaching

Preserving your mannequin dimension in verify might in flip maintain your infrastructure prices in verify. Right here are some things you may be mindful whereas getting your massive NLP mannequin prepared.

1. Take into account the out there assets: The scale of the LLM for deployment ought to take note of the out there {hardware} assets, together with reminiscence, processing energy, and storage capability. The LLM’s dimension ought to be throughout the limits of the out there assets to make sure optimum efficiency.
2. Tremendous-tuning: Select a mannequin with optimum accuracy after which fine-tune it on a task-specific dataset. This step will enhance the effectivity of the LLM and maintain its dimension from spiralling uncontrolled.
3. Take into account the tradeoff between dimension and efficiency: The LLM’s dimension ought to be chosen based mostly on the tradeoff between dimension and efficiency. A bigger mannequin dimension could present higher efficiency however can also require extra assets and time. Due to this fact, it’s important to seek out the optimum steadiness between dimension and efficiency.

Use a light-weight deployment framework

Many LLMs are too massive to be deployed on to a manufacturing surroundings. Think about using a light-weight deployment framework like TensorFlow Serving or TorchServe that may host the mannequin and serve predictions over a community. These frameworks might help cut back the overhead of loading and working the mannequin on the server thereby lowering the deployment and infrastructure prices.

Put up-deployment mannequin monitoring

Mannequin monitoring helps optimize the infrastructure price of deployment by offering insights into the efficiency and useful resource utilization of deployed fashions. By monitoring the useful resource consumption of deployed fashions, akin to CPU, reminiscence, and community utilization, you may determine areas that may provide help to optimize your infrastructure utilization to scale back prices. 

• Monitoring can determine underutilized assets, permitting you to reduce on unused assets, and lowering infrastructure prices. 
• Monitoring can determine resource-intensive operations or fashions, enabling organizations to optimize their structure or refactor the mannequin to be extra environment friendly. This will additionally result in price financial savings. 

Key takeaways

Conclusion

On this article, we explored the challenges we face when deploying an LLM and the inflated infrastructural price related to them. Concurrently, we additionally addressed every of those difficulties with the required methods and options. 

Out of all of the options we mentioned, a few issues that I might advocate probably the most with regards to lowering infrastructure price whereas deployment is elastic and serverless inference. Sure, mannequin compression is sweet and legitimate, however when the demand is excessive, even the smaller mannequin can act like a bigger mannequin, thus rising the infrastructural price. Thus, we have to have a scalable strategy and pay-per-demand service. That’s the place these inference providers get useful. 

It goes with out saying that my advice won’t be probably the most perfect in your use case, and you may decide any of those approaches relying on the form of issues you’re coping with. I hope what we mentioned right here will go a good distance in serving to you chop down your deployment infrastructure prices in your massive NLP fashions. 

References

1. Large Language Model Training in 2023
2. https://d1.awsstatic.com/events/Summits/reinvent2022/AIM405_Train-and-deploy-large-language-models-on-Amazon-SageMaker.pdf
3. Top 10 AI Chip Makers of 2023: In-depth Guide 
4. https://www.nvidia.com/en-us/data-center/dgx-a100/
5. LLaMA: A foundational, 65-billion-parameter large language model
6. https://arxiv.org/pdf/2203.15556.pdf
7. https://huggingface.co/docs/transformers/model_doc
8. https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast
9. https://sunniesuhyoung.github.io/files/LLM.pdf
10. https://twitter.com/tomgoldsteincs/status/1600196995389366274?lang=en
11. https://arxiv.org/pdf/1910.02054.pdf
12. https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
13. Jaime Sevilla et al. (2022), “Estimating Coaching Compute of Deep Studying Fashions”. Printed on-line at epochai.org. Retrieved from: ‘https://epochai.org/blog/estimating-training-compute‘ [online resource]
14. https://arxiv.org/abs/2001.08361
15. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf
16. https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
17. https://aws.amazon.com/sagemaker/neo/
18. https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/7126bf7beed4c4c3a05bcc2dac8baa3c/pruning_tutorial.ipynb
19. https://towardsdatascience.com/distillation-of-bert-like-models-the-code-73c31e8c2b0a
20. https://aws.amazon.com/blogs/machine-learning/train-175-billion-parameter-nlp-models-with-model-parallel-additions-and-hugging-face-on-amazon-sagemaker/
21. Improving Language Model Behavior by Training on a Curated Dataset
22. https://towardsdatascience.com/how-to-deploy-large-size-deep-learning-models-into-production-66b851d17f33
23. https://huggingface.co/blog/large-language-models
24. https://aws.amazon.com/blogs/machine-learning/deploy-large-models-on-amazon-sagemaker-using-djlserving-and-deepspeed-model-parallel-inference/
25. Large Language Models Can Self-Improve
26. https://spot.io/resources/cloud-cost/cloud-cost-optimization-15-ways-to-optimize-your-cloud/
27. https://dataintegration.info/choose-the-best-ai-accelerator-and-model-compilation-for-computer-vision-inference-with-amazon-sagemaker
28. https://medium.com/data-science-at-microsoft/model-compression-and-optimization-why-think-bigger-when-you-can-think-smaller-216ec096f68b
29. https://medium.com/picsellia/how-to-optimize-computer-vision-models-for-edge-devices-851b20f7cf03
30. https://huggingface.co/docs/transformers/v4.17.0/en/parallelism#which-strategy-to-use-when
31. https://medium.com/@mlblogging.k/9-libraries-for-parallel-distributed-training-inference-of-deep-learning-models-5faa86199c1f
32. https://towardsdatascience.com/how-to-estimate-and-reduce-the-carbon-footprint-of-machine-learning-models-49f24510880