Table of Contents
A local LLM is a large language model running on a company computer or server rather than in a provider’s cloud. It makes sense when a company works with data it does not want to send externally, uses AI every day and has someone on the team who can maintain the environment. If none of these conditions apply, cloud services and ready-made subscription tools will be sufficient.
More and more companies implementing artificial intelligence combine both approaches: they handle sensitive data and repetitive tasks locally, while using the cloud for everything else. This article explains how to calculate the costs, what a local model does and does not change for data security, and where to start.
What is a local LLM?
A local LLM (local large language model) is a language model whose files and compute run on company hardware: a laptop, desktop computer, workstation or server (known as on-premises AI). When a company runs an LLM locally, prompts, documents and responses do not go to an external API.
Most often, these are open-weight models, for example from the Llama, Mistral, Qwen or Gemma. They can be downloaded and run without a charge for each query. Licence terms vary between models, so they should be reviewed before commercial use.
A local model can operate fully offline. AI without cloud services or internet access matters wherever security is critical, for example in manufacturing plants or isolated IT environments. The broader term “local AI” also includes models for image and speech recognition, as well as image generation.
Cloud, subscription or your own hardware: a comparison
No option is best at everything. The decision depends on the data a company processes and how intensively it uses AI.
| Criterion | Cloud (API) | Subscription | Your own hardware |
|---|---|---|---|
| Where data goes | To the provider’s servers | To the provider, in line with the agreement | Stays within the company |
| Cost model | Usage-based billing (tokens) | Fixed fee per user | Hardware purchase, then power and maintenance |
| Model quality | Most capable models | Models selected by the provider | Open-weight models, including large ones |
| Tool integration | Via API; requires development | Ready to use, in the company’s applications | In-house or with a partner |
| Scaling | Immediate | By purchasing additional licences | Limited by hardware capacity |
| Operation without internet access | No | No | Yes |
| Who maintains the environment? | Provider | Provider | The company’s IT department or a partner |
Open-weight models vary in size. Within the same family, larger variants often deliver better results, but parameter count alone does not determine quality. Smaller models are sufficient for day-to-day work. Large models running on computers with NVIDIA DGX Spark, workstations or servers with GPU cards can approach leading cloud models in selected business tasks.
Your own hardware reduces dependence on an external provider, but application and infrastructure security then become the company’s responsibility.
How much does a local LLM cost and when does it pay for itself?
A local LLM is most likely to pay for itself when usage is high and consistent. For only a few queries a week, buying hardware is usually harder to justify financially than using the cloud.
Cloud and subscription costs rise with usage: with every additional team member, every agent test and every reprocessing of a document base. The cost of your own hardware is incurred mainly upfront, while subsequent queries primarily generate electricity costs and, for servers, cooling costs as well. Hardware depreciation, maintenance and administrator time are fixed costs included in the calculation below.
How to calculate it
- Add up annual cloud AI expenditure: per-user licences, API charges, and the costs of testing and development environments. Include what the company plans for the coming year.
- Assess which of these tasks can be moved to a local model, then calculate what they currently cost in the cloud. Tasks requiring the most capable models will remain in the cloud.
- Add the cost of your own environment: hardware, power, cooling, administrator time and maintenance support.
- Spread the hardware price over its useful life, for example 36 months, and compare the monthly cost of your own environment with the monthly cost of the tasks you will move from the cloud (point 2). If it is lower and the response quality is sufficient, a local LLM is worthwhile.
What has the biggest impact on the outcome?
- Usage. Hardware that sits idle generates costs without delivering benefits. An agent operating for several hours a day or a team testing models daily shortens the payback period the most.
- Number of users. A single sufficiently powerful device can support an entire team, depending on the model, the number of concurrent queries and the deployment approach.
- Experimentation. In the cloud, every attempt, such as a new prompt version or rebuilding a document index, comes at a cost. Locally, you can test without a meter running.
- Skills. If nobody in the company can maintain the environment, the cost of a partner must be included.
Data security: what a local LLM solves and what it does not
A local model eliminates one risk: sending data to an external provider. However, it does not remove the need to comply with GDPR or the AI Act.
What the company gains
- Documents, source code, financial data and customer data remain within the company’s infrastructure. This is particularly important where professional or banking confidentiality applies, such as in law firms, healthcare providers and banks.
- The company decides where query logs are stored and for how long.
- Employees have fewer reasons to use unapproved private AI tools, whose data retention and usage practices may not comply with the company’s security policy.
- The model can operate on a network that is completely isolated from the internet.
What local deployment does not solve
- GDPR. If the solution processes personal data, a legal basis for processing, access controls for personal data and the ability to delete it are still required. With a local deployment, you must also ensure access is logged.
- AI Act. Obligations depend on the company’s role and how it uses AI, not on where the model runs. From 2 August 2026, transparency obligations apply, including informing users that they are interacting with an AI system and labelling deepfake content. Requirements for high-risk systems have been deferred until December 2027 and August 2028.
- Permissions and AI agents. The model must respect the same permissions as the company’s other systems; otherwise, it will show every user all documents that it can access. AI agents that open files and run tools autonomously should have additional permission restrictions, as they may execute instructions hidden in document content (known as prompt injection).
- Updates. Models, libraries and the system require updates just like any other software.
What companies use local models for
Local LLMs work best for tasks involving confidential data and high query volumes.
Assistant for company documents (RAG)
The model answers questions in natural language based on procedures, contracts, manuals and the knowledge base. Documents are not sent to an external service. The response can reference the source document, allowing employees to verify where the information came from immediately.
AI agents
Programs that analyse files, prepare reports or handle tickets can run for hours without per-query charges. Local deployments are also not subject to an external provider’s API limits. However, performance is constrained by available hardware resources, so longer tasks are best run outside team working hours, when full hardware capacity is available.
Working with code
A developer assistant works with source code that cannot leave the company. It takes over repetitive work: suggesting code snippets and preparing tests and documentation. Developers still design solutions, review every AI suggestion and decide what is included in the project; they can use the time saved for more complex tasks. The model can also be adapted to the company’s codebase, such as its proprietary libraries and coding standards, so that suggestions align with the team’s practices.
Document processing
Classifying, summarising and extracting data from invoices, orders and complaints. With thousands of documents per month, cloud token charges rise fastest for these tasks.
Model fine-tuning
Adapting a model to the industry’s language, product names and company processes using the organisation’s own data. Training data—often among a company’s most valuable knowledge assets—remains on its own hardware.
Images and video
Image recognition for quality control, and generating graphics and video content without sending projects externally. On a production line, local processing also reduces response times because images do not need to travel to the cloud and back.
These solutions are most commonly used by software houses, R&D departments, financial companies, manufacturing businesses, e-commerce companies and organisations with stringent security requirements.
When the cloud still makes sense and how to combine both approaches
The cloud is well suited where access to specific models, rapid scaling or ready-made integration with everyday applications matters:
- tasks best suited to specific cloud-based models, such as complex analysis and multi-step reasoning,
- occasional use that does not justify purchasing hardware,
- sudden demand spikes, such as one-off processing of a large archive,
- assistants built into office suites, such as Microsoft 365 Copilot, which work with email, calendars and files within existing permissions.
Hybrid model
A hybrid model allocates tasks according to data type and required quality:
- Sensitive data (code, financial documents, customer data): local LLM.
- High-volume repetitive tasks (classification, summaries, agents running around the clock): local LLM.
- Everyday office work in company applications: subscription-based assistant.
- The most demanding tasks involving non-sensitive data: the most powerful cloud models.
A hybrid model is one option, not a ready-made formula. The right allocation of tasks depends on the type of data, scale of use and the team’s expertise, so it is worth reviewing it periodically.
What to run a local LLM on
The selection below is dominated by hardware based on NVIDIA solutions. CUDA is one of the best-supported AI acceleration ecosystems. Popular tools such as PyTorch, Ollama and llama.cpp support NVIDIA GPUs, although many also support other platforms. NVIDIA also offers hardware for every scale, from laptops to servers, with a consistent tool ecosystem.
A new addition to Senetic’s range is NVIDIA RTX Spark™ laptops. They combine NVIDIA Blackwell RTX graphics and a NVIDIA Grace processor in a single chip, delivering up to 1 PFLOP of AI performance (FP4), and run Windows 11. The key specification is unified memory ranging from 24 to 128 GB, shared by the processor and graphics. This determines how large a model can be run locally. In the highest configuration, the laptop can run models with up to 120 billion parameters and a context of up to 1 million tokens.
The thinnest designs are 14 mm thick and, according to NVIDIA, the laptops offer all-day battery life. However, the claimed battery life does not mean all-day intensive workloads with a large model, as it depends on how the device is used. Applications run locally, while cloud resources can be used when needed, which fits the hybrid model well. Senetic’s range includes Microsoft Surface Laptop Ultra, ASUS ProArt P16, HP OmniBook Ultra 16, Lenovo Yoga Pro 9n and MSI Prestige N16 Flip AI+.
Hardware selection is primarily determined by memory, as the model must fit into it. We covered the requirements and specifications in our guide on which computer to choose for AI, and infrastructure investment levels in the article on types of servers for AI.
Hardware type should be matched to the scale and nature of the tasks:
| Use case | Hardware type | Where to find it in our range |
|---|---|---|
| One person, including when travelling | NVIDIA RTX Spark laptop (up to 128 GB) | AI laptops |
| Shared AI environment for a team | Computer with NVIDIA DGX Spark | AI computers |
| Fine-tuning and the largest models | Workstation with NVIDIA DGX Station | AI workstations |
| Multiple concurrent users | Server with GPU cards | AI servers |
| Expanding existing hardware | GPU card for a server or workstation | AI GPU cards |
Where to start
- Choose one use case, such as an assistant working with documents from a single department.
- Check which data the model will process and who should have access to it.
- Test several open-weight models on real tasks, for example using tools such as Ollama or LM Studio, and compare the results with a cloud model.
- Estimate how many queries per day the use case will generate, then calculate the costs using the four steps described above.
- Check software compatibility with the hardware. Computers with NVIDIA DGX Spark and NVIDIA RTX Spark laptops use Arm-architecture processors, so not every application will run on them without modification.
- Based on the tests and calculations, choose the type and number of devices to match the model sizes and number of users.
The return on investment in local AI hardware depends primarily on how extensively it is used. It is therefore worth preparing a list of specific use cases before making a purchase.
Frequently asked questions
Does a local LLM work without an internet connection?
Yes. Once the model and tools have been downloaded, they can be used entirely offline. A connection is required only for updates, downloading new models and any integrations with external services.
Is a local model as good as cloud models?
For everyday tasks such as summarisation, classification or answering questions based on documents, open-weight models are often sufficient. For more complex tasks, the result depends on model size: large open-weight models running on workstations or servers come close to leading cloud models. It is therefore worth comparing them using your own tasks.
Is local AI automatically GDPR compliant?
No. A local model limits the transfer of data externally, but the company remains responsible for the legal basis for processing, data access and data deletion.
How much memory does a local LLM need?
This depends on the model size and its level of compression (quantisation). With 4-bit compression, each billion parameters requires approx. 0,5–0,6 GB of memory, plus additional capacity for the conversation context and system. In practice, a model with 8 billion parameters requires approx. 5–6 GB, one with 70 billion requires approx. 40 GB, and one with 200 billion requires more than 100 GB. Without compression, requirements are roughly four times higher.
Will a local LLM replace Microsoft 365 Copilot?
No. They are tools for different tasks. Copilot works with email, files and calendars in Microsoft 365. A local model is suitable for data that should not be moved to the cloud and for a company’s own AI applications. Many organisations use both.
Can a project running on local hardware be moved to a data centre or the cloud?
Yes, provided it uses the same tools, such as CUDA and containers. However, the processor architecture must be considered: NVIDIA DGX Spark and NVIDIA RTX Spark laptops run on Arm, while many servers run on x86, so some libraries and container images may need to be prepared for the other architecture.
