On-premise AI for SMEs: when it pays to keep AI in-house, and what it costs

Does it pay for an SME to keep AI in-house? Yes, for the data that must not leave: trade secrets, documents with personal data, the knowledge that makes the difference. For everything else, in my experience, the cloud remains cheaper and more capable. The practical path is hybrid: a small local model works on sensitive data, the cloud reasons about the rest.

My thesis is that after twenty years of cloud first, AI is coming back inside the company. Not instead of the cloud, but next to it. At GTC Berlin on 21 October 2026, Jensen Huang, who leads NVIDIA, explained that a company can take an open model and further train it on its own data, turning it into what he called "sovereign intelligence"; and that the knowledge and tools of agents belong to the company (NVIDIA's official live blog). The person saying it sells the machines in-house AI would run on: a good reason not to take it literally. But it is the same argument we have heard for years about Europe depending on others for energy and chips, the base of AI, brought down to the level of a single company.

What does on-premise AI mean for an SME?

On-premise AI means the model runs on a computer the company controls: a server on site, a workstation, sometimes a PC. The documents you give it never leave the perimeter, and no outside provider receives them. It is the opposite of a cloud subscription, where every request travels to someone else's servers.

For an SME it is almost always about small models, called SLMs (small language models): reduced versions that run on ordinary hardware and do one specific job well. Which model to pick does not depend on the flag of whoever publishes it, but on measured results for the task you need.

When does it pay to keep AI in-house, and when not?

Privacy is a recurring brake among companies that consider AI. According to Istat, the Italian national statistics institute, among companies with at least 10 employees that did not use artificial intelligence in 2025 but had considered using it (11.5% of non-users), 43.2% listed privacy and data protection among the obstacles (Istat, Enterprises and ICT, 15 December 2025). A local model reduces exactly that brake.

It pays when:

• you work on trade secrets: formulas, recipes, quotes, projects, anything a competitor would pay to read;
• you handle personal data of customers, employees or debtors: keeping it in-house avoids handing it to an outside provider, even though the other GDPR obligations remain (I wrote about it in the piece on AI agents that find customers and GDPR);
• you want to protect the company's knowledge base and use it much more, without depending on a provider that can change prices and terms.

It does not pay when the task needs the best reasoning available, when volumes are low, or when nobody in the company can keep one more machine running. There the cloud wins today, and for me it remains the lever to cut costs and improve quality and processes.

What can a small local model already do?

Three jobs I have run with small models on my PC: a laptop with an NVIDIA GeForce RTX 4060 graphics card with 8 GB of video memory, not a dedicated machine.

Anonymizing documents before handing them to an AI. It is the process we worked with for LidIA, the platform for over-indebtedness procedures developed by Castaldo Solutions in collaboration with Rimedia Solutions (I tell how it was born in the case study). On my PC a small model anonymized the documents, and only the cleaned document went to an AI to digest. It is the hybrid model in its simplest form: sensitive data never leaves, and the reasoning is done by whoever does it best.

Reading documents (OCR). Turning scans and PDFs into text you can search and use. It is repetitive, high-volume work on documents that often should not leave: the typical case for a small model. On my laptop I did it with GLM-OCR, a 1.1-billion-parameter model that takes 2.2 GB and downloads for free with Ollama.

Generating images from other images. An img-2-img agent for the prototype of an architecture firm: you start from an image of the project and generate variants, without sending the drawings to an outside service.

One thing needs saying clearly: today none of these three jobs runs on a dedicated machine at a client. My clients are SMEs with tight margins, and so far dedicated hardware has not been justified. But they work, and this is where you start.

How much does hardware for local AI cost?

To try it, if you already have a computer with a good graphics card, you do not need to buy anything: free tools such as Ollama or LM Studio download an open model and run it locally. A laptop with 8 GB of video memory already runs small models for OCR, anonymization and images.

When you need a dedicated machine, NVIDIA's smallest reference is the DGX Spark, a desktop computer designed to run models locally. US list prices in dollars, excluding taxes, checked on 24 October 2026:

For larger systems I did not find a public price list, and the table does not include running costs: energy, maintenance, the time of whoever looks after it.

The right comparison is between the cost of the machine and what you already pay in subscriptions and working hours on that task. And there is a part the numbers do not see: if local AI protects the company's secrets and frees it from services you do not know what they will do with your data, for me that price is priceless.

What is a harness, and why does it matter more than the model?

In the Berlin keynote an AI agent is described as the sum of five components:

• the model;
• the harness;
• the sandbox, the isolated space where it acts;
• knowledge;
• tools.

The harness is everything around the model that makes it work: the instructions, the tools it can use, the knowledge it can read, the permissions, the checks. It is the part that makes the difference, and it is our job: building all of this around the model, that is, being agentic harness implementers.

The model gets swapped when a better one comes out; the harness stays, and with it the rules and the company's memory. Instructions are the first piece, and a concrete example is the CLAUDE.md file. It holds locally too: a small model with clear instructions, tight permissions and well-organized knowledge does its job well. The same model without a harness is a demo.

Where should an SME start?

With a list, not a purchase. Write down which data must not leave the company, and on which of it AI would save you the most time. Pick one task, for example anonymizing one type of document, and try it locally on a PC. Only when the use is clear does it make sense to choose the hardware, and at that point you do the math on numbers, not on fear.

If you want to understand which of your data should stay in-house and which can go to the cloud, you start from a free Pre-Assessment: write to me, it is a conversation worth having before you buy anything.

Frequently asked questions

What is on-premise AI?

It is artificial intelligence that runs on hardware controlled by the company, on site or on a dedicated machine, instead of on a cloud provider's servers. The documents and data you give it never leave the company perimeter. For an SME it usually means small models that run on ordinary hardware and do one specific job well, such as anonymizing or reading documents.

How much does it cost to run AI locally?

To try it, nothing: free tools such as Ollama or LM Studio run an open model on a computer with a good graphics card. For a dedicated machine, NVIDIA's DGX Spark launched at $3,999 and rose to $4,699 in February 2026; in October 2026 the 64 GB Dell Pro Max with GB10 costs $4,999. The cost should be compared with the subscriptions and hours that task costs today.

Is a small local model as good as ChatGPT or Claude?

In my experience, not on long, open-ended reasoning, where frontier cloud models stay ahead. On narrow, repetitive tasks, yes: anonymizing documents, reading scans and generating variants of an image are jobs a small model does well on your own PC. That is why the practical path is hybrid: local works on sensitive data, the cloud reasons about the rest.

Is local AI enough to comply with GDPR?

No. A local model avoids handing personal data to an outside provider, and with it the agreement GDPR requires when a third party processes data on your behalf (Article 28). But the company still carries the legal basis for processing, the information notice to the people involved and the security measures (Article 32). The full text is in Regulation (EU) 2016/679.