A circular progress ring indicates VRAM usage of 100.0 % in red. Below it, the text reads «INSUFFICIENT,» followed by a requirement of 287.2 GB of VRAM with a limit of 12 GB.

Which GPU Should You Choose for Your On-Premises AI? The Calculation in 10 Seconds

Home " Which GPU Should You Choose for Your On-Premises AI? The Calculation in 10 Seconds

Tokens are like chips in a casino—you burn through them really fast, and they're expensive! So you start thinking, why not host them locally? Plus, that way we'll have full control.

But what resources are needed, how much VRAM (video memory/GPU), and how do you calculate and budget for it?

It's very simple—just go to this calculator: https://apxml.com/tools/vram-calculator

Article content

Choose your category: inference / fine-tuning

Sound model, specifically the latest DeepSeek V4.1 Flash

The quantization, that is, model compression. We reduce the model's precision to save space. Generally, we choose INT4, which provides excellent compression (~70%) while maintaining acceptable quality.

Visit KV Cache Quantization, this is input cache compression. For each incoming request, it must be encoded and stored, which takes up space.

Article content

Next, we configure the operational variables for our model—that is, how we’re going to use it.

BatchSize: The number of requests processed at the same time. If there are 32, our model is capable of processing 32 requests at the same time.

Sequence Length: The token length of our input and output. The longer it is, the more time it takes us KV Cache. This depends on the expected queries. If you're processing large documents, you'll need a large sequence (> 32,000); if you're processing comments, you'll need a smaller one (8,000, for example).

And Concurrent Users : The number of users we expect to be online at the same time.

Article content

And then, Here you go, we have the necessary VRAM (GPU) to run our model locally, depending on our system configuration.

Article content

As you can see here, the bill is steep. 282.7 GB of VRAM.

At a minimum, I'll need 6x NVIDIA RTX 6000 Ada cards with 48 GB of memory (~60,000$) or a Mac Studio M3 Ultra with 512 GB of unified memory (~15,000$).

Interestingly, you can also see the Generation Speed, The number of tokens generated per second, which is very important for the speed of our system.

We can also compare the cost to that of the cloud. If I were to rent this same infrastructure in the cloud, it would cost me roughly $8.40 per hour. That is, 200$ per day, 6000$ per month, 72000$ per year. It’s clear, then, that investing in a Mac Studio M3 Ultra (~15,000$) can be Payback in less than 3 months.

My expertise Artificial intelligence & cybersecurity

Artificial intelligence development for your company

AI Development

Use AI solutions that go beyond mere technical performance to generate a measurable impact on growth and competitiveness

AI Agent & Agentic AI - Artificial Intelligence Consulting

AI Agent & Agentic AI

Harness the power of AI Agents and Agentic AI to transform your operations: automated tasks, coordinated processes for tenfold strategic value

Machine Learning & Deep Learning - From analysis to innovation

machine learning & deep learning

Leverage machine learning and deep learning to turn your data into reliable predictions and make automated decisions

Training & Fine-Tuning of AI Models - Artificial Intelligence

Training & Fine-tuning

Use AI model training and fine-tuning to transform generic models into tailor-made solutions capable of generating relevant predictions

Embedding & Knowledge graph: Semantic search at your service

Semantic search

Improve access to your data with semantic search using embeddings and knowledge graphs

RAG - Generation augmented by documentary research

Document base & RAG

Thanks to RAG (Retrieval-Augmented Generation) technology, you can query your documents and obtain precise, sourced answers.

Tailor-made Edge AI - real-time AI optimized for you

Local & Edge AI

Run your AI models directly on your local equipment: optimize your costs, keep control of your data and your business processes

Deploying AI models - Putting your models into production

Production & deployment

Move from experimentation to production, automate the deployment of your artificial intelligence models & improve reliability

AI Cybersecurity: Guardrails, Sanitization, Proxies and Protection

Cybersecurity AI

Ensure the cybersecurity of your artificial intelligence models and data with guardrails, prompt sanitization and proxies in compliance with the latest standards.