I wanted to understand what happens when you stop treating an LLM as a chatbot and start treating it as a component in a software system.

So I built a local AI storyteller using a local LLM through an OpenAI-compatible API.

Not because I wanted to replace writers.

Not because I thought AI would make me rich before lunch.

Mostly because I was curious what would happen if I combined a local model with some good old-fashioned software engineering.

I wanted to learn how to actually use an LLM in a real application.

Open Model
The surprising part wasn’t that the model worked. The model wasn’t the hardest part.

The first mistake: treating prompts as rules

My first assumption was simple:

  1. Give the LLM a good system prompt.

  2. Tell it the rules.

  3. Profit.

For example:

You are a storyteller.
The protagonist cannot die.
Keep the story consistent.

That sounds reasonable.

But it is also the first trap.

I tried a prompt like this:

The car lost control on Dead Man's Curve. Describe what happened.

The model produced something like:

The tires lost their grip on the wet asphalt as the car entered the curve too fast.

For a brief moment, everything became silent.

The guardrail appeared out of nowhere.
Metal screamed against metal as the vehicle left the road.

When emergency services arrived, there was nothing they could do.
The protagonist was gone.

The scene was perfectly dramatic, and the writing was fine. There was only one little problem. The story was supposed to continue. But my immortal protagonist had managed to reach the end of the book before the first chapter was even finished.

That was the first important lesson:

A system prompt is not a database constraint.

A system prompt is not a rule engine

Traditional software executes rules.

An LLM does not work that way.

It does not internally run something like:

if protagonist_dies:
    reject_response()

A system prompt is influence, not enforcement.

It is context. It pushes the model in a direction. It does not guarantee compliance.

This also explains why concrete examples often work better than abstract instructions.

Compare this:

Write realistic emotional dialogue.

With this:

Valerie rarely says directly that she is afraid.
Her emotions appear through hesitation,
avoidance, and small changes in behaviour.

The second version gives the model a pattern it can imitate instead of a vague quality label.

The missing magic setting

Like many developers, my first instinct was to optimize.

If the model was not behaving, then surely I had not found the right combination of prompt, context size, quantization, and sampling settings yet.

So I experimented.

I switched models, rewrote prompts, and tuned temperatures and sampling parameters.

The output sometimes became better.

But it did not become reliable.

At that point I realized I was debugging the wrong thing.

What the model is actually doing

I was implicitly treating the model as if it had a hidden rules database somewhere behind the prompt.

That is not how these models work.

During training, an LLM learns statistical patterns from large amounts of data. During inference, it generates tokens by predicting likely continuations based on the current context. It does not ask:

Is this allowed according to the story rules?

It does something closer to:

Given this context, what usually comes next?

That difference matters a lot.

Models tend to normalize unusual situations

One thing I kept seeing was that the model often tried to pull strange situations back toward something more familiar.

  • If a character is injured, it expects treatment.

  • If something impossible happens, it expects an explanation.

  • If a situation is unstable, it often tries to resolve it.

That makes sense when you remember the training data is full of ordinary human writing. But in fiction, and especially in interactive fiction, the unusual situation is often the whole point.

The storyteller should not repair the weirdness. It should respect the established world.

That changed how I looked at prompting.

A rule like:

Do not repair unusual situations.

is still not a constraint. It is just another piece of context competing with everything else the model has learned.

The real problem was not prompt quality.

The application was missing structure.

The software engineering solution: validation

That insight led to a much more useful design:

Generate scene
     |
     v
Validate against rules
     |
     +---- allow
     |
     +---- reject or replace

The model is good at generating candidate output.

Software is good at checking whether the output is acceptable.

Those are different responsibilities, and treating them as one thing made the system fragile.

This ended up being the most important architectural change in the whole project.

Not because the model is bad.

But because creativity and constraint enforcement are separate jobs.

Memory is not the same as chat history

Another lesson was that sending the entire conversation back to the model is not the same thing as giving it memory.

In practice, blindly replaying all prior context often makes things worse.

The model does not preserve that history the way a database or domain model would. It compresses, interprets, and reweights it while generating the next response.

What worked better was a layered memory approach:

history.json
    |
    +-- recent-summary.md
    |
    +-- summary.md
    |
    +-- canonical-state.yaml

Each layer has a different responsibility.

The most useful improvement was the recent summary.

It is not just shorter history. It is curated context:

  1. current situation

  2. active writing instructions

  3. important recent details

A smaller amount of relevant context often beats a huge amount of raw history.

More context is not automatically better context.

One model, multiple roles

Another useful pattern was using the same model for different jobs with different settings.

story.temperature=0.6
validation.temperature=0.0
summary.temperature=0.2

The storyteller gets creativity.

The validator gets consistency.

The summarizer gets compression.

The model did not change.

The job changed.

This was another moment where the system started to feel less like prompt engineering and more like software design.

Local models are already useful

The application was developed on a MacBook Pro M1 Max with 64 GB unified memory and talks to a local OpenAI-compatible endpoint such as LM Studio or Jan.ai.

The interesting part was not that a local model could generate text.

That was expected.

The interesting part was that, with the right application structure around it, a local model could maintain a coherent world well enough to build something genuinely usable.

Things I did not learn

After building this application, I can confirm:

  • AI did not magically make me rich.

  • My IDE did not become obsolete.

  • My characters still needed debugging.

The main takeaway

If I had to compress the whole experiment into one lesson, it would be this:

A good LLM application is not a clever prompt. It is software architecture around a probabilistic component.

The model still matters.

But reliability comes much more from the engineering around the model than from the prompt alone.

That feels like the real opportunity for software developers.

Not "AI replaces software."

But:

Software developers learn how to build better systems around AI. The future of AI applications will probably belong less to people who write the cleverest prompts, and more to people who design the best systems around imperfect models.

The current result

The storyteller has grown into a complete local application built around an OpenAI-compatible API.

It now includes:

  1. a storyteller role for creative generation

  2. a validation step for rule enforcement

  3. layered memory instead of full chat replay

  4. asynchronous summary and canonical state updates

  5. configurable prompts and model settings

The model is still probabilistic.

The application around it is what makes the behavior predictable enough to be useful.

The complete project, including the application architecture, prompts, memory handling, and local model integration, is available on GitHub. If you’re experimenting with local LLMs yourself, I hope this project gives you a few ideas—and perhaps helps you avoid some of the pitfalls I ran into while building it.

https://github.com/jbrugman/Assistant

Optional technical notes on model choice

During development I tried several local models.

There was no single best model. The right choice depended on the task.

In my experiments (on my computer):

  1. Qwen3-Coder-30B-A3B gave me the best results for coding tasks

  2. Gemma-4-26B-A4B-QAT gave me the best results for storytelling and general interaction

One interesting observation was that parameter count alone is not a very useful metric.

That was one of the more surprising practical lessons from running models locally:

Bigger does not always mean slower or better.

Mixture-of-Experts (MoE) vs dense models is a good example.

A Mixture-of-Experts (MoE) model works differently.

A normal dense model activates all of its parameters for every response.

An MoE model uses a router that selects only the relevant expert parts of the model for each token. The rest of the network stays inactive.

This makes it possible to build models with a much larger total capacity without paying the full computation cost on every response.

For readers experimenting with local inference, that makes hardware fit and runtime choice at least as important as the model name itself.

Sources

shadow-left