Mid-sized companies face different challenges in AI initiatives than large enterprises. Budgets are smaller, teams leaner, and expectations swing between "this solves everything" and "this is just hype" — often within the same meeting. What is needed is an architectural framework that combines pilot speed with production stability.
The typical mistake: betting on hyperscalers too early
Many first AI projects fail not on model quality but on running costs. A RAG application generating 5,000 euros per month in inference fees rarely survives the first budget review. On top of that comes the data outflow to the US — a no-go for many B2B customers in regulated industries.
The pragmatic start instead runs on self-hosted, scalable infrastructure: a Kubernetes cluster (on-premise or with a European provider), an object store such as MinIO for documents, a Postgres with pgvector for embeddings, and a gateway layer that can mediate between a local model and a cloud fallback.
Four components that have to work together
The stack that has proven itself in our projects consists of four building blocks: an indexer that reads documents from MinIO, chunks them, and writes them as embeddings into pgvector. A gateway that bundles model requests, enforces rate limits, and implements fallback logic. An orchestrator that selects the right agent depending on the request — chat, search, catalog, download. And a frontend widget that can be embedded in existing websites via iframe.
The separation is not academic. Each of these components evolves at a different pace, and each has its own security and compliance profile. A monolithic approach forces them all into the same cost model and release cycle.
From pilot to production
The transition from pilot to production often fails on one question: "Who runs this in everyday operations?" Without a clear answer, the project ends up with whoever built it — and turns into technical debt with the first staff change.
The clean answer is a GitOps pipeline with ArgoCD: every change to the system is a commit, every release a tag, every rollback a git revert. That dramatically lowers the requirements on the operations team and makes the system independent of individual people.
What comes next
In the next post, we'll show how this architecture was applied in a concrete customer project — including the obstacles we did not see coming.