From Native to Cloud: Estimating GPU Sources for Open-Supply LLMs | by Maxime Jabarian | Nov, 2024
Estimating GPU reminiscence for deploying the newest open-source LLMs
In case you’re like me, you most likely get excited in regards to the newest and best open-source LLMs — from fashions like Llama 3 to the extra compact Phi-3 Mini. However earlier than you soar into deploying your language mannequin, there’s one essential issue you’ll want to plan for: GPU reminiscence. Misjudge this, and your shiny new internet app would possibly choke, run sluggishly, or rack up hefty cloud payments. To make issues simpler, I clarify to you what’s quantization, and I’ve ready for you a GPU Reminiscence Planning Cheat Sheet in 2024— a useful abstract of the newest open-source LLMs available on the market and what you’ll want to know earlier than deployment.
When deploying LLMs, guessing how a lot GPU reminiscence you want is dangerous. Too little, and your mannequin crashes. An excessive amount of, and also you’re burning cash for no motive.
Understanding these reminiscence necessities upfront is like figuring out how a lot baggage you may slot in your automobile earlier than a street journey — it saves complications and retains issues environment friendly.
