Project 07 / Case study
Self-hosted infrastructure · AI
A server of my own serves a large model through an OpenAI-compatible API. It is not there as decoration: the CRM, the client scrapers, the RAG demo and a game with an AI mode all talk to it through one environment variable. This page is about why that comes out cheaper than a token bill, and what is paid in exchange.
Every one of my projects that uses AI is paid for by tokens. A scraper that classifies thirty thousand company records is not a query, it is an invoice, and the margin of the product goes into that. Then there is the other cost, the one that does not show on the bill: to enrich a client database I have to send that database to a third-party server. I wanted to reason with large models with no meter, and without handing a single piece of client data over to anyone.
An inference service in a container that exposes an OpenAI-compatible API. The model arrives quantised and spread across the available memory. Each project points at it by changing one environment variable, exactly as it would point at a commercial provider: the code does not know where the model that answers it is.
That a purchase is not judged on the spec sheet but on the cost per task completed. And that the hard part is not bringing it up: it is deciding which model goes in, at what precision, how many layers are offloaded to main memory, and what happens when the model does not answer. That last one I already had solved in another project, and here it became the important thing again.
Architecture
The compute server runs the heavy services in containers. The other machine acts as the front door: each service stops being an IP with an odd port and gets a domain instead, with its TLS, from one single point.
The base is a machine of my own dedicated to compute, which spends the day with whole models mapped into memory. I chose the hardware for the memory available and not for the speed, and that conditions everything that comes after in this page.
That is the reason this project exists: a model with 125 billion parameters, even quantised to three bits, does not fit on a single consumer card. With a single one you would have to serve a model four times smaller, and a model four times smaller is another product. The GGUF format also allows the surplus to be offloaded to main memory, so there is room to raise the amount of context without touching the model.
On top of that run several services in containers, each with its own independent data volume. The main one is the GPU inference server with an OpenAI-compatible API: a single endpoint that can be called with any client that already exists. The rest are development and automation services that consume that same endpoint. They live here and they are not published.
The second machine solves a different problem. A service on 192.168.x.x:8000 is not something you can explain to anyone, and as soon as you move a container to another port everything that depends on it breaks. With a reverse proxy that gives a domain to each service, the projects point at a stable name and the internal mapping changes without anyone noticing. The separation between the two machines is deliberate: restarting the compute server to try something should not bring down the front door, or the other way round.
Decisions
Every one of them has a rejected alternative and a cost that is paid every month.
The cards are from a generation already retired for new workloads. With that purchase a lot of graphics memory is chosen for a fraction of what it costs to match it in the current generation, and the rest is given up: the acceleration features that current cards bring as standard, and a manufacturer support that is on its way out. It is a trade, not a bargain. To serve a large model to one user or two, memory rules and the generation is secondary; to serve a lot of people at once, this machine is not the answer and it will not be.
All of my projects talk to models through the same format as always, which is the one the commercial provider made popular. Serving through that same format means that changing provider, or going back to the cloud for a test, is changing a URL and a key. The CRM proves it: its entire integration sits behind a flag and a base address, and it can run without AI without touching one line. The cost is depending on a third party's format, which they change whenever they feel like it, and losing the parameters that belong to each engine and do not travel down that pipe.
The model is served quantised, in a format that also comes split across several files. Lowering the precision of the weights is what makes a model with 125 billion parameters fit into the available memory with room for the context. The bill is paid in quality on very fine tasks: in classifying company records, or in extracting an email address from a web page, it is not noticeable; in a long and ambiguous text, it is. The honest comparison is not against the model in full precision, it is against the small model that would fit without quantising, and there the quantised version wins by a distance.
There is a service that works with logged-in sessions, and it is the most delicate one on the machine: whoever reaches it handles everything you have open. It is deployed but stopped, and it only comes up when there is a run. Zero attack surface on the days it is not used. The cost is that every run starts with a cold start, a few seconds that go unnoticed in a batch lasting hours.
What it gives and what it costs
The good is real and so is the bad. I will put both columns.
What it gives. Zero marginal cost per token: a batch that classifies thirty thousand records costs the same in compute as a single question, and that changes which things are worth automating. No rate limits, so a scraper can go at its own pace without a provider cutting it off halfway. The data does not leave the house, which for a client database is not a technical detail but the condition for being able to work with it. And the model that I choose is served, with no terms of use over what may be done with the output and no content filters that appear when a provider changes a policy. No subscription fee, no card, no depending on a third party not deciding to withdraw a version.
What it costs. The hardware is paid for up front and degrades in silence: if tomorrow I want to serve a model with 200 billion parameters and long context, the answer is another purchase, not a credit card. The electricity consumption is permanent. And the operation is mine: nobody calls me when something falls over, so if a restart leaves a service down I find out when a project fails. There is no monitoring set up here yet, and that is real debt, not a virtue.
What I do not believe, and that is why I say it. I have not measured tokens per second, nor sustained concurrency, nor temperature with the room shut. The figures on this page come from code and from use, and there is not one single performance measurement because I do not have it yet. Whoever reads this page and wants a performance number, I cannot give it to them yet.
First, measure before upgrading. I built the service and put it to work without ever recording tokens per second, card utilisation or the real electricity cost per batch. Without that baseline, the next upgrade is a hunch. Second, monitoring and alerting: when a service falls over at night nobody notices until a project fails, and that has already happened. Third, review how the cards communicate with each other: with no direct link declared, everything goes through the board's bus, and as soon as there are simultaneous requests that will be the bottleneck. And what comes next, exposing tools to the agent so that the machine does not only answer with text but acts on the services already running on it: that is the reason I set the compute up at home instead of renting it.
I do not publish addresses, domains, credentials or the content of the private repositories that live on this machine, nor the specifications of the hardware or what runs inside it. What is told here is the decision and its cost, which is what can be reasoned about. What I have not measured is marked as not measured.