I rented a stack of H200s and ran GLM-5.2 myself. The small models had already won.

A small business I know wants AI that’s genuinely useful, but their customer data can’t leave the building. That single rule quietly kills the obvious answer. You can’t pipe sensitive records through a cloud API, no matter how good it is.

So the question becomes: can an open model, running on hardware the business actually owns, do the job? And before anyone spends money on a GPU, can I find out cheaply?

I decided to rent the hardware and test it. And I want to be upfront about something, because it became the whole point by the end: renting a cloud GPU does not give you privacy. Your data still lands on someone else’s machine in someone else’s data centre. The rental was never the solution — it was a way to de-risk the buy decision. Run the models, measure speed and quality, then decide what’s actually worth owning.

Here’s what happened.

The test

I wrote seven deliberately awkward prompts — the kind that separate a model that looks clever from one that is: spot the bug in a function, implement a rate limiter, a multi-step word problem, follow a strict “JSON only, no prose” instruction, pull fields out of a messy note, refuse to hallucinate a spec it wasn’t given, and write to a tight word count with no bullet points.

I ran them against ten models, raw — no agent harness, no tools, no retries. I wanted to measure the model, not a clever wrapper around it:

• The frontier cloud ladder — Opus 4.8, GPT-5.5, Gemini 3.1 Pro at the top, down to Sonnet 4.6 as my “floor” (the lowest quality I’d accept).
• Three small open models that fit a single 24GB GPU via Ollama — qwen3.6:27b, gemma4:26b, qwen3-coder:30b.
• And the big one: GLM-5.2 — a 743-billion-parameter open model, the best open weights you can get right now.

Then I had two different AIs grade every answer independently — Claude Opus and GPT-5.5 — because one judge is just one opinion.

What surprised me: the small models were already good enough

This was the headline before I even got to the giant. On these tasks, a free open model running on a single second-hand £790 RTX 3090 matched the quality floor of a paid frontier API. gemma4:26b and qwen3.6:27b scored at or above Sonnet 4.6 — and the data never left the room.

The losses among the small models weren’t about being stupid. They were about polish: one model invented a confident-sounding statistic; another wrapped perfect JSON in a code fence and broke a strict format rule. The gap between a good small open model and a frontier model has genuinely narrowed to instruction-following discipline, not intelligence.

And the cost? Renting that 3090 and running all three small models through all seven prompts came to about 18 pence.

Hold that number.

Then I downloaded a 743B model and ran it myself

I wanted to feel the ceiling. So I rented the hardware to self-host GLM-5.2.

The first attempt failed completely. I deployed eight A100 GPUs, and the model never served a single answer — a chain of problems ending in a fatal driver/CUDA mismatch. I’d spent about £17.50 to produce nothing. (Lesson one: even getting the giant to load is a real engineering job.)

So I did the thing I’d tell anyone to do — I stopped guessing. I put the plan to a panel of eight different AI models for debate, and did a grounded research pass against the actual documentation and a real-world benchmark someone had published. The verdict corrected my plan: GLM-5.2’s exotic attention kernel only runs properly on newer “Hopper” GPUs, so 8× A100 was the wrong box. Switch to 4× H200.

Second attempt: it worked. And here’s what self-hosting the giant actually looked like:

• It needed 4× H200 GPUs at $17.56/hour — secure-cloud only, because nobody had them spare on the cheap tier.
• The model is 410 GB. You download all of it before it says a word.
• Single-user speed was about 8 tokens/second — and one prompt took 958 seconds. The model spiralled, talking to itself for nearly 8,000 tokens, and never produced the answer.
• One run of seven prompts cost $24.86.

Remember that 18 pence — same test rig, same seven prompts.

The twist in the score

Raw, GLM-5.2 came last — 64%. I almost wrote that down as the result. But it’s a trap, and both judges flagged it.

GLM-5.2 is a “thinking” model, and because of how I served it, it was the only model that dumped its internal reasoning straight into its answers. The grader then marked the whole think-out-loud scratchpad. That broke the strict-format prompts (a 900-word preamble before a perfectly valid JSON array still fails “JSON only”). When I stripped the leaked reasoning and re-graded the actual answers, GLM-5.2 jumped to 93%.

So it’s genuinely capable. But here’s the honest part: even cleaned up, it still didn’t beat the free small models — because of that one prompt it couldn’t stop reasoning about long enough to answer. The 743-billion-parameter ceiling didn’t buy a clear win on practical work. It bought operational complexity.

If you try this yourself — the lessons

A few things I learned the hard way, in case you’re tempted:

• The data-centre GPU is built for throughput, not latency. 8 tokens/second for one user sounds slow, but that’s me using a 50-lane motorway with one car on it. The same box would serve dozens of people at once. I was stress-testing the dimension this hardware is worst at.
• One config flag tripled my cost. I’d added a setting (--enforce-eager) to save a minute at startup; it disabled the thing that makes the engine fast and slowed every token by ~3×. Small knobs, big bills.
• Serving a model well is a skill in itself. Reasoning parsers, streaming to dodge timeouts, format handling — get them wrong and a brilliant model produces a mess. The raw 64% vs cleaned 93% was that gap.

The conclusion that actually matters

Truly private is not renting a cloud server. I keep coming back to this. The reason to run an open model yourself is so the data never leaves your control — and the moment you rent a GPU in someone else’s data centre, it has. The rental was a test rig, not the destination.

If privacy is the point, privacy comes from owning the box. And the surprising, genuinely useful finding is that you don’t need a giant for it:

• For real privacy-first day-to-day work — drafting, extraction, Q&A, coding help — build the ~£1,600–2,100 “Gemma-4-on-a-3090” machine. It matches the cloud floor, it owns its data, and it costs pennies to run. (That’s the £790 card plus a sensible box around it.)
• For occasional top-tier reasoning on non-sensitive work, just use a metered frontier API. It’s cheaper and faster than owning the ceiling.
• The frontier-open giant? Don’t self-host it for normal business use. If you genuinely need it, hit its API for a few pence. Self-hosting it is a specialist project — research, open-weights control, high-throughput serving — not a small-business tool.

A note on honesty

I won’t oversell the numbers. I didn’t trust a single AI’s marking, so I had two different ones — Claude and GPT-5.5 — grade every answer independently. They agreed on what actually matters: the small open models match my floor, and the giant isn’t worth owning. I’ll also own that a couple of my own setup choices made GLM-5.2 look worse than it really is — a properly tuned serve would score higher. The full method, both graders’ marks, and the complete failure log are published, so anyone can check my working.

Own small. It beats renting huge.