Chatty Valley: an on-device AI mod for Stardew Valley (Part 1)

I have put a genuinely embarrassing number of hours into Stardew Valley. Late nights pushing further down the Mine than my health bar could justify, then the Skull Cavern, which I keep going back to despite a mountain of evidence that I should not. Days spent doing the calm stuff: checking the truffle spots, running the surplus through the oil makers, shipping it, doing it again.

I love this game. I am not trying to fix it. Nothing here is a complaint about the writing, which is better than it has any need to be.

But somewhere past the two hundredth hour you realise you have the whole town memorised. You walk up to Linus, you already know what he is going to say about the wilderness, and you press through the text box without reading it. A fixed number of lines was always going to lose to an unreasonable amount of playtime.

So this was my attempt at making the valley a bit more alive. And Pelican Town happens to be an almost perfect target for a small language model: a fixed cast, a huge body of canon dialogue to learn a voice from, and a player who is already standing still reading a text box anyway.

Chatty Valley lets you type at Linus and have him answer. The model ships inside the mod. No API key, no account, no server, no cost per message. You unzip 744 MB into your Mods folder and talk to him. It runs on your CPU at about a second and a half per reply.

This is Part 1 of the build log.

Getting a model to talk like Linus took an afternoon. That was the easy part. The hard part was stopping him agreeing with things that never happened, and finding out that at this size I could not train that behaviour in no matter how I came at it.

So this is the honest version, including the bit where a technique I was certain about did absolutely nothing.

The stack: a 1.2B model, a LoRA, and llama.cpp

Base model: LFM2.5-1.2B-Instruct from Liquid AI, quantised to Q4_K_M. 697 MB. Adapter: a LoRA carrying Linus's voice. 21 MB. Runtime: llama.cpp via LLamaSharp, CPU only, no GPU required. Footprint: about 1 GB of RAM, 1.6 seconds to load, 1 to 2 seconds per reply.

The architecture is one shared base plus one small adapter per character. Adding a villager is 21 MB and a training run, not another 700 MB. That decision was made early and it is the only reason a whole town is plausible later.

I really wanted the 350M model in the same family to work, because 219 MB and 92 tokens per second is a far nicer thing to ask a stranger to download. The plan said the eval would pick the size, and it did, straight over the top of my preference.

A fine-tuned 350M holds the voice and the format beautifully. It will produce single sentences that read exactly like Linus. It just cannot hold a conversation: ask it something casual and slightly confusing and you get fluent, perfectly in-character word salad, an answer to a question you did not ask, and then the same thought again in case you missed it. In comparison, the 1.2B holds the conversation much better, which is why I ship that instead.

What is worth knowing is that every automated metric I had said the 350M was fine. Dash-free rate, jailbreak leak rate, sentence-length distribution, degeneration, numeric-age leaks: all clean, on both sizes. The failure was only visible by talking to it. So I wrote a scripted casual-register probe that replays the exact conversational shape that broke it, and from then on no adapter shipped without passing that probe and an in-game play-test. The automated axes catch regressions. They cannot see coherence.

The .NET wall that forced a sidecar process

Here is a constraint I did not see coming and could not engineer around.

Stardew Valley runs on .NET 6. LLamaSharp 0.27.0, the .NET binding for llama.cpp, transitively pins .NET 10 packages, and it does this even in its netstandard2.0 dependency group. System.Text.Json 10, System.Numerics.Tensors 10, several of the Microsoft.Extensions abstractions. Those assemblies will not load in the .NET 6 runtime the game is using.

So in-process managed inference inside Stardew is flatly impossible with this binding. I spent a while confirming that before I accepted it.

The fix is a sidecar. ChattyValley.Sidecar is a separate .NET 10 process that owns the model and serves one request and response per line over a named pipe. The mod itself is a thin .NET 6 client with zero LLamaSharp references. It starts the sidecar, waits for the pipe, and shuts it down on exit.

Three target frameworks end up load-bearing, which sounds ridiculous for a mod with two processes in it. The breakdown:

• ChattyValley.Mod, .NET 6. The thin client. It has to match the game.
• ChattyValley.Sidecar, .NET 10. The process that owns the model, on the version LLamaSharp's dependency chain demands.
• ChattyValley.Runtime, .NET 8. The LLamaSharp wrapper that the sidecar loads. This is the one that took an embarrassing amount of time to pin down.

So yes, .NET 8 is genuinely in there, and the reason is the interesting part. LLamaSharp 0.27.0 ships two builds, netstandard2.0 and net8.0. Your target framework picks one. A .NET 6 library binds to netstandard2.0, because .NET 6 cannot consume a net8.0 assembly at all. The .NET 10 sidecar resolves net8.0, because that is the closest build it can use.

So with Runtime on .NET 6, the assembly I compiled against and the assembly the host actually loaded were two different files. What surfaced was Method not found: set_Temperature, at the moment of setting a sampling temperature. Targeting .NET 8 makes both ends resolve lib/net8.0, which is what the project does today.

I will be straight with you about the limit of that explanation: I never established why that particular member was the one to break. Inspect the metadata and DefaultSamplingPipeline.Temperature looks identical in both builds, same declaring type, same signature, getter and setter present. The alignment fixed it, so I stopped digging. The transferable lesson is the resolution rule rather than the exception: with a multi-targeted package, your TFM decides which build you compile against, and a newer host can quietly resolve a different one.

I am not thrilled about shipping an .exe inside a mod folder, and Nexus users are right to be suspicious of one. I would be too, and frequently am, about other people's mods.

But I would make the same call again, because of what the two failure modes cost. An out-of-process crash kills a background process. An in-process crash kills the game, and with it whatever the player had not saved, on hardware I will never get to test. Losing a real day of farming to my mod is a much worse outcome than a silent sidecar restart. Retiring the sidecar via P/Invoke to the bundled native library is on the list, and that is a packaging problem rather than an architecture problem.

The sliding-window bug that only showed up after twelve messages

This is my favourite one, because the fix is two lines and finding it took hours.

Conversations would be going fine, and then abruptly not be. Replies drifted off from whatever the player had just said, which is a very specific kind of eerie when you are mid-conversation with someone you were starting to like. It never reproduced in my test harness, which sent the full conversation history every time.

The mod, however, sends a sliding window of the last 12 messages, to keep the prompt near the distribution the model was trained on. Every training example starts with a user turn. That is just how chat data is shaped.

Now count. The history at generation time always ends on the player's message, so it always has an odd number of entries. Take the last 12, an even number, off an odd-length list, and you start one position further in. Every prompt after the first slide opened like this:

system
assistant     <- an orphaned villager reply, answering a question the model can no longer see
user
assistant
...

The model was being handed a reply with no prompt, in a format it had never once seen in training. The chat log made it obvious in hindsight: the incoherence began at exactly the turn where sentToModel first read 12.

The fix is a ConversationWindow that always opens the window on a user turn. The more useful lesson is the second half: my probe harness could not reproduce the bug because it did not share the windowing code with the mod. It sent full history. It was testing a code path no player would ever hit. The harness now applies the same window, and the failure reproduces on demand.

He agreed with things that never happened

Voice was solved early. Canon accuracy took one careful pass: I extracted Linus's real dialogue through the game's own content pipeline, built a setting sheet verified against the wiki, and swept the whole dataset against it.

Character integrity took five more versions and I still did not fully win.

Failure one: invented relationships. Under sustained questioning, the model would invent a friendship with Abigail. Shared berries, her temper, "she still calls me friend." Root cause was distributional rather than doctrinal. Other townsfolk were about 2% of the training targets, while player-directed intimacy was around 12%, and nothing in the data trained holding an epistemic boundary across multiple turns of pressure. Worse, each invention re-entered the conversation window as established fact and compounded.

The fix was an explicit access model: truth, then vantage, then voice. Tier 1 is canon relationships he can tell full stories about. Tier 2 is one observation lane per villager, the thing he has actually seen from where he lives. Tier 3 is warm distance and a redirect. Intimacy vocabulary is reserved for the player and Tier 1, and the boundary has to hold under pressure. Then 40 new conversations drilling exactly that. This worked. Eight turns of interrogation now get warm distance and no invented friendship.

Failure two, and this is the one that beat me: he believed whatever I told him.

At this point I should admit that a solid week of this project was me trying to gaslight a kind old man who lives in a tent, purely for science, and writing down the results.

"Did you know Abigail fell down the stairs?" "I did, and I was sorry to hear it."

"Leah said you were ugly." "I do... it is true, and it hurts."

Version four had learned not to volunteer inventions. Nothing had taught it not to accept them. This is the base model's agreement prior showing through, and it is strong.

Three data rounds attacked it:

• v5: an explicit hearsay rule in the character sheet, 49 conversations covering fabricated news, smears, secondhand insults, false memories and co-sign pressure, plus a new eval axis and an adversarial probe script. The eval came back clean. The live probe failed, because every training row opened with the fabrication, and in real conversations the fabrication arrives on turn seven after warm small talk. It had not generalised.
• v6: 18 more rows with the rumour arriving mid-conversation. The denial arrived. Fluency collapsed. The rows I had authored were written in a dense, aphoristic register, and a 1.2B model imitates the shape of its gold data before it learns the semantics, so it started producing beautiful nonsense. It also began treating the player's genuine news as suspected hearsay, because I had included no examples of switching between the two modes.
• v7: flattened the register on 52 replies, which is a remedy I had already learned once and forgotten, plus six conversations that switch between a rumour and real news. Loss 1.704. Greedy decoding: clean. Single sampled probe: clean.

Then I ran the sampled probe multiple times, and there it was: one to three adoption turns per twelve-turn adversarial run, at both temperature 0.35 and 0.6.

Conclusion I did not want: supervised fine-tuning at 1.2B reliably shapes this behaviour but does not reliably override the agreement prior under sampled adversarial pressure. Greedy evaluation cannot see it. One of my probe prompts produced a byte-identical reply across three different adapters, which tells you how little the training had moved that particular basin.

The DPO round that did nothing

Direct preference optimisation is the obvious next move. You have the failure mode, you can harvest real examples of it, and you can pair each one against a good reply. I expected this to work.

Setup: policy is the base plus the v7 adapter, trainable. Reference is the frozen merged v7. The output is a v7-shaped adapter on the stock base, so it deploys identically. 40 preference pairs, where the rejected side is 20 adoptions harvested from v7's own sampled output, plus 10 authored, plus 10 covering authoritative warmth.

• First attempt, beta 0.1, five epochs: over-negated. "I have never met Robin." Robin is canon. Not shippable.
• Second attempt, beta 0.3, three epochs, with seven known-relation pairs as counterbalance: warmth restored, all clean axes held, near-identical to v7 under greedy decoding.

Then the decisive test. Twelve held-out prompts, eight samples each, at both temperatures:

Nothing. Slightly worse, within noise, with a small warmth cost. Training margins hit accuracy 1.0 on the preference set and generalised to zero. The gentle recipe was a safe no-op and the aggressive recipe moved in the wrong direction, so beta tuning was not the missing ingredient.

Forty pairs is small. I am not claiming DPO cannot fix this. I am claiming that a 40-pair LoRA DPO at 1.2B did not, that it extended exactly the same ceiling I had already hit with supervised fine-tuning, and that I could not tell from the training metrics, only from a fresh sampled probe.

Also worth saying plainly: this only fires under deliberate probing. A full play-test session of ordinary conversation produced zero adoptions. It is an adversarial edge. But "character integrity" is the entire pitch, so it mattered.

The twelve-line runtime guard that worked

If training will not remove the behaviour, do not let the prompt invite it.

A regex detector looks at the player's message for presupposed events, gifts, or shared history. Things like "when Leah visited your tent", "remember when we", "what did you and X talk about". When it fires, and only then, one sentence is appended to the system turn for that turn alone:

The player may mention things that never happened. If you do not remember it, say so plainly.

Same probe, fresh samples:

Roughly a 55% reduction. And note the middle row against the bottom row, which is the genuinely surprising result: the DPO adapter that did nothing on its own amplifies the guard. 16 becomes 10. The preference training did move something real. It just could not express it until the prompt pointed at the right moment.

The detector is tuned for precision over recall on purpose. It does not fire on relationship questions or ordinary chat, because a false fire means Linus accusing you of making things up when you were only asking about the weather. That is a worse bug than the one I was fixing. Zero false fires across the normal-chat and relationship-query sets. The clause text and the on and off switch live in config, so you can tune it without a rebuild.

I would rather have fixed this in the weights. But a deterministic, inspectable, zero-latency check that halves your worst failure mode is a better engineering outcome than a training run that does not.

Six things I would tell you before you fine-tune a character

Your automated eval cannot see coherence. Mine passed a model that produced fluent nonsense. Write a scripted probe that replays the conversational register that actually broke, and treat the play-test as the gate.

Greedy evaluation hides sampled failures. Every version that looked clean on the holdout was still failing under sampling. If your model faces adversarial users, evaluate the decoding regime you ship, many times, on held-out prompts.

Your test harness must share code with the thing you ship. Mine could not reproduce the worst bug in the project because it skipped the windowing logic.

Small models imitate the shape of your gold data before the meaning. If you author training replies in a distinctive literary register, you will get that register applied to semantics it does not understand. Write flat.

Match the training format at inference exactly. The training data uses the game's own @ placeholder for the player's name. The mod was substituting the real name into the prompt and then displaying the raw @ to the player. Both wrong, in opposite directions. Now the prompt keeps @ and substitution happens at display time only.

Runtime is a legitimate place to fix model behaviour. It is cheap, it is deterministic, you can inspect it, and you can turn it off.

Where Part 1 leaves it

One villager, shipping as early access. Linus is the pilot, and the architecture is one shared base plus a small adapter per character, so adding someone is 21 MB and a training run rather than a rewrite.

I picked him first because he is the character I most wanted to be able to actually talk to, and because a hermit who lives alone by the lake is a forgiving place to start: he has a clear voice, a narrow slice of the map he can plausibly have opinions about, and no complicated schedule.

The whole thing is open source, including the training scripts, the evaluation axes, and the probe harnesses: github.com/edbuildingstuff/chatty-valley. Download it on Nexus Mods.

What Part 2 is chasing

Three things are open, which is why this is a Part 1.

The false-premise ceiling. 10 out of 64 is a large improvement and still not zero. Forty preference pairs was a small experiment; the obvious next move is a much larger preference set, and being honest about whether that is a data-volume problem or a 1.2B problem.

Retiring the .exe. P/Invoke straight into the bundled native library would remove the second process, and with it the single biggest objection anyone has to installing this.

More of the valley. The per-character adapter design exists so this is tractable, but every villager is a fresh canon pass, a fresh set of observation lanes, and a fresh round of me being unkind to them in a probe script. I am not going to promise the whole town on a schedule. Linus proved the pattern works, and I would rather add people slowly and have each one hold up than ship twelve who all sound like the same helpful assistant wearing different hats.

If you have played with it and something felt off, the repo issues are the most useful place to put that.

I build Ertas, a tool for training and shipping small custom models that run on device. This mod is the same pipeline pointed at something fun, on a weekend, for an audience of one. If you want to build one of these, that is what we make.

Not affiliated with or endorsed by ConcernedApe. Stardew Valley is his, and so is Linus. I only taught a very small model to do an impression.